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FOREWORD 


The  U.S.  Army  has  made  a  substantial  commitment  to  the  use  of  simulations  for  training 
and  readiness.  Many  current  simulators  are  networked  and  designed  to  provide  realistic  training 
and  rehearsal  for  large  combined  arms  groups  of  vehicles  and  major  weapon  systems.  These 
simulators  represent  dismounted  Soldier  activities,  but  are  not  intended  to  directly  train  or 
rehearse  individual  dismounted  Soldiers.  Virtual  Environment  (VE)  technology,  which  typically 
includes  head-mounted  visual  displays  with  tracking  devices  for  limbs  and  individual  weapons, 
provides  increasing  capabilities  that  enable  a  more  immersive,  person- centered  simulation  and 
training  capability  for  dismounted  Soldiers.  These  systems  are  being  investigated  in  order  to 
include  individual  dismounted  Soldiers  in  the  larger  networked  simulation  systems,  and  to 
support  training  and  rehearsal  for  dismounted  Soldiers.  One  research  challenge  arising  from 
these  efforts  is  identifying  and  quantifying  the  effects  of  VE  system  characteristics  and  use  on 
learning,  retention,  and  transfer  of  skills  required  for  Army  tasks. 

This  report  describes  one  experiment  in  an  ongoing  program  of  research  conducted  by  the 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI),  Simulation  Systems 
Research  Unit  (SSRU)  that  addresses  the  use  of  VE  technology  for  training  dismounted  Soldiers 
in  simulations.  This  experiment  investigated  the  effects  of  different  supplemental  stimuli  for 
directing  attention  and  providing  task  guidance  during  repeated  performance  in  virtual 
simulation  exercises.  The  findings  from  this  research  will  be  used  to  recommend  VE 
characteristics  and  instructional  methods  for  incorporation  in  distributed  VE  simulation  systems 
for  training. 

SSRU  conducts  research  with  the  goal  of  providing  information  that  will  improve  the 
effectiveness  of  training  simulators  and  simulations.  The  work  described  here  is  a  part  of  ARI 
Research  Task  233,  VICTOR  -  Virtual  Individual  and  Collective  Training  for  Objective  Force 
Warrior.  The  initial  results  from  this  work  were  presented  at  the  Interservice/Industry  Training, 
Simulation,  and  Education  Conference  2004. 


Acting  Technical  Director 


v 


ACKNOWLEDGMENT 


This  research  could  not  have  been  conducted  without  the  excellent  programming  and  technical 
support  provided  by  the  Institute  for  Simulation  and  Training  (1ST)  at  the  University  of  Central 
Florida.  We  especially  thank  Glenn  Martin  and  Jason  Daly  of  1ST  for  their  creative  efforts  in 
producing  the  virtual  environment  models  and  data  collection  software  used  in  the  experiment. 


vi 


INSTRUCTIONAL  FEATURES  FOR  TRAINING  IN  VIRTUAL  ENVIRONMENTS 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  is  committed  to  using  interactive  simulations  for  mission  planning, 
rehearsal,  training,  concept  development,  and  test  and  evaluation.  Many  current  simulators  are 
designed  to  provide  training  for  Soldiers  fighting  from  vehicles,  but  are  not  designed  to  provide 
realistic  training  on  multiple  tasks  for  dismounted  Infantry.  Virtual  Environment  (VE)  and 
gaming  technology  is  providing  new  capabilities  for  simulating  real  world  activities  for 
individual  dismounted  Soldiers.  This  technology  may  allow  the  U.S.  Army  to  cost-effectively 
conduct  planning,  training,  and  rehearsal  activities  for  both  individual  and  collective  dismounted 
Soldier  tasks.  Basic  to  these  simulations  is  the  common  context  for  individual  combatants  who 
need  to  move,  observe,  shoot,  and  communicate.  Accepting  as  a  given  the  baseline  of  adequate 
functional  and  physical  fidelity  of  equipment  and  environment  for  the  performance  of  Soldier 
tasks,  the  capabilities  of  VE  technology  can  go  much  further.  The  technology  can  support  or 
provide  stimuli  to  the  learner  that  could  enhance  learning  through  the  use  of  training  or 
instructional  strategies,  tactics,  and  the  addition  of  instructional  features.  Research  on  the  effects 
of  specific  VE  system  characteristics  and  instructional  applications  must  be  performed  in  order 
to  establish  the  benefits,  problems,  and  guidelines  for  training  and  rehearsing  complex  activities 
and  tasks  using  VE  technology. 


Procedure: 

This  experiment  was  developed  to  investigate  the  training  effect  of  instructional 
interventions  in  the  use  of  VE  technology  during  training  of  representative  Soldier  tasks.  We 
used  previous  analyses  to  select  the  representative  tasks,  and  a  defining  structure  is  suggested  for 
a  hierarchy  of  instructional  approaches,  leading  to  instructional  interventions  that  should  apply  to 
the  representative  tasks.  The  experimental  training  protocol  is  designed  to  examine  the  effects  of 
the  presence  or  absence  of  two  factors:  Interrogative  Coaching  and  an  Attention-Direction 
Instructional  Feature,  on  the  initial  skill  acquisition  performance  of  dismounted  Soldier  tasks  that 
incorporate  basic  recognition  and  decision  skills.  The  tasks  addressed  were  Bounding 
Overwatch  movement,  Helicopter  Landing  Zone  identification  and  marking,  and 
Observation/Fire  Post  positioning  (around  the  landing  zone).  The  instructional  factors  were 
investigated  using  a  relatively  simple  2x2  experimental  design,  allowing  repeated  exercises  to  be 
used  for  monitoring  initial  skill  acquisition.  Participants  were  recruited  from  a  local  university, 
and  compensated  for  their  time  in  the  experiment.  Participants  trained  on  the  tasks  using  an 
instruction  booklet,  and  conducted  an  exercise  incorporating  basic  instruction  and  practice  on 
operations  in  the  VE  before  beginning  the  experiment.  During  the  experiment  participants 
worked  through  three  training  exercises  with  appropriate  experimental  instructional 
interventions,  and  participated  in  After  Action  Reviews  (AARs)  following  each  exercise.  A 
fourth  “test”  mission  without  intervention  or  AAR  was  then  conducted. 


vii 


Findings: 


Data  were  collected  on  the  rated  performance  of  the  three  tasks,  the  time  to  perform  the 
tasks,  the  number  of  buildings  inspected  during  the  tasks,  and  the  amount  of  time  taken  to  inspect 
buildings  during  the  tasks.  The  performance  data  was  sufficiently  non-normal  that  the  ratings 
were  transformed  before  analysis.  Analyses  revealed  significant  improvement  over  the  repeated 
exercises  in  both  time  and  performance  on  the  Bounding  Overwatch  and  Landing  Zone  tasks, 
indicating  that  participants  learned  (improved  performance  in  less  time)  during  the  training 
exercises.  Both  the  Interrogative  Coaching  and  Attention-Direction  Arrow  Instructional  Features 
significantly  interacted  with  the  repeated  missions  in  the  Bounding  Overwatch  task.  The  data 
show  that  the  Coaching  intervention  improved  performance  relative  to  the  non-coached 
condition,  but  the  Attention-Direction  aided  group  performed  worse  over  trials  than  those  not 
receiving  the  intervention.  Analysis  of  the  Landing  Zone  performance  times  found  the  Coaching 
intervention  significantly  interacting  over  missions,  with  the  intervention  leading  to  slower 
performance  times.  Analysis  of  the  fourth  mission  only  found  significant  improvements  in 
performance  on  the  Bounding  Overwatch  task  from  the  Coaching  intervention. 


Utilization  and  Dissemination  of  Findings: 

The  results  of  this  initial  investigation  indicate  that  there  is  no  initial  learning  advantage  to 
the  type  of  Attention  Direction  feature  that  was  used,  and  that  the  application  may  have 
interfered  with  initial  skill  acquisition.  The  Interrogative  Coaching  did  seem  to  aid  the  skill 
acquisition  of  the  more  complex  Bounding  Overwatch  task,  although  it  also  extended  the  time  to 
perform  the  Landing  Zone  task.  These  results  indicate  that  instructional  interventions  are 
difficult  to  successfully  implement  in  an  experimental  setting.  The  results  are  also  being  used  to 
shape  continued  investigations  into  the  use  of  instructional  strategies,  tactics,  and  features  in  VE 
simulations  for  dismounted  Soldier  tasks.  The  next  experiment  will  address  different 
characteristics  in  both  the  delivery  and  format  of  coaching  during  practice  performances,  and 
alterations  in  visual  stimuli  for  Attention-Direction  information  presentation  (non-normal,  task- 
specific  stimuli  that  enhance  or  call  attention  to  normal  environmental  task  cues).  The  results  of 
this  and  succeeding  experiments  can  and  should  inform  the  design  and  development  of 
simulation  environments  and  training/rehearsal  applications  for  U.  S.  Army  Soldiers. 
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INSTRUCTIONAL  FEATURES  FOR  TRAINING  IN  VIRTUAL  ENVIRONMENTS 


Introduction 

U.S.  Army  Soldiers  are  tasked  with  a  growing  array  of  complex  and  challenging  tasks  and 
missions.  Dismounted  Infantry  Soldiers,  in  particular,  must  possess  knowledge  and  skills  that 
are  instantly  accessible  to  survive  and  excel  in  warfare  operations  today.  One  aspect  of  the  U.S. 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  mission  is  to  enhance 
individual  Soldier  and  group  performance  along  with  group  and  individual  decision  making 
(ARI,  2004,  2005).  Information  supporting  improvements  in  training  for  Soldiers  through  the 
use  of  simulation  technology  is  a  main  thrust  of  the  Simulation  System  Research  Unit  (SSRU),  in 
supporting  the  ARI  mission. 


Simulation  Research  and  Application 

Simulation  has  long  been  proven  effective  for  training  new  skills  and  allowing  Soldiers  to 
practice  these  skills  in  an  interactive,  dynamic  fashion.  Older  examples  include  flight  simulators 
(Williges,  Roscoe,  &  Williges,  1973)  and  tank  simulators  (Alluisi,  1991).  Newer  examples 
include  the  Virtual  Sand  Table  -  a  computer-generated  version  of  the  traditional  sand  table 
exercise  which  gives  personnel  the  opportunity  to  practice  military  doctrine  in  a  hands-on 
manner  (Wisher,  Macpherson,  Abramson,  Thornton,  &  Dees,  2001),  and  more  immersive 
systems  that  allow  Soldiers  to  navigate  through  virtual  settings  with  helmet  mounted  displays 
(HMDs)  and  body-movement  tracking  (Knerr,  et  al.,  2003). 

The  continual  increase  in  use  of  Virtual  Environment  (VE)  simulation  can  be  attributed  to 
the  numerous  benefits  that  simulation  provides  in  comparison  to  traditional,  real  world 
experience  training  programs  (Hays  &  Singer,  1989;  Swezey  &  Andrews,  2001).  First,  as  with 
the  earlier  vehicle-based  simulation  systems,  VEs  afford  greater  flexibility  to  precisely  alter 
learning  environments  and  mission  conditions.  An  example  of  this  flexibility  is  the  ability  to 
implement  instructional  strategies  and  features  that  are  difficult  or  impossible  to  implement  in 
real  world  settings.  Second,  they  can  provide  superior  performance  assessment  capabilities, 
although  additional  processing  may  be  needed  for  interpretation  of  data  (Banta,  Troillet, 
Hefferman,  Plamondon,  &  Beal,  2004).  For  example,  VE  systems  are  well  suited  to  capture  and 
store  detailed  performance  data  (e.g.  Knerr,  et  al.,  2002).  Third,  evidence  shows  that  the 
effectiveness  of  VE  training  for  some  tasks  is  equivalent  to,  or  in  some  cases  better  than,  real 
world  training  (Rose  et  al.,  2000;  Todorov,  Shadmehr,  &  Bizzi,  1997).  Finally,  it  is  obvious  that 
simulation-based  training  is  safer,  as  emergencies  can  be  presented  and  emergency  procedures 
practiced  in  a  simulator  that  would  never  be  allowed  during  the  operation  of  real  equipment 
(Williges,  Roscoe,  &  Williges,  1973).  In  summary,  VE-based  simulation  will  continue  to 
increase  in  use  and  usefulness  for  training  many  tasks  and  skills.  This  has  already  been 
demonstrated  in  training  a  variety  of  tasks  or  jobs,  for  example  with  pilots  (e.g.  Bell  &  Waag, 
1998;  Lintem,  Roscoe,  Koonce,  &  Segal,  1990);  ship  navigators  (Hays  &  Vincenzi,  2000; 
Magee,  1997);  emergency  personnel  (Bliss,  Tidwell,  &  Guest,  1997);  first  responders  to  bio¬ 
terrorism  (Stansfield,  Shawver,  Sobel,  Prasad,  &  Tapia,  2000);  and  space  mission  ground  control 
staff  (Loftin,  Wang,  Baffes,  &  Hua,  1992). 
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ARI’s  SRU),  contractually  supported  by  the  University  of  Central  Florida’s  Institute  for 
Simulation  and  Training  (1ST),  established  a  research  program  in  VE  technology  in  order  to 
investigate  a  wide  range  of  potential  VE  training  issues  and  factors  for  training  individuals  and 
teams.  One  research  and  development  goal  is  to  “develop  and  evaluate  procedures  and 
techniques  for  using  embedded  training,  low-cost  simulation,  and  augmented  reality  to  enhance 
the  fighting  capabilities  of  digitally  equipped  dismounted  individual  Soldiers  and  leaders”  (pg.  6, 
U.S.  Army  Research  Institute,  2004).  Earlier  research  and  analysis  focused  on  the  VE 
requirements  and  application  guidelines  for  leader  and  individual  performance  in  unit  tasks 
(Jacobs,  et  al.,  1994),  the  determination  of  necessary  characteristics  for  VE-based  individual 
combatant  training  (Knerr,  1998),  and  the  evaluation  of  potential  transfer  of  training  to  military 
operations  (Knerr,  et  al.,  2003).  More  recent  work  evaluated  the  feasibility  of  training 
individuals  in  a  shared  virtual  space  while  geographically  distributed  (Singer,  Grant, 
Commarford,  Kring,  &  Zavod,  2001)  and  communication  modality  and  After  Action  Review 
(AAR)  performance  in  a  distributed  immersive  VE  (Kring,  2004).  The  latter  work  follows  a 
more  general  approach,  that  of  conducting  research  with  non-military  (untrained  on  military 
tasks)  participant  populations  including:  research  aids,  student  interns,  and  university  students. 
This  approach  has  the  benefit  of  easing  the  acquisition  of  participants  for  general  training  issues 
research,  and  the  drawback  of  not  using  a  more  representative  population  sample  (e.g.  Army 
Soldiers).  - 

The  next  step  in  this  program  of  research  is  to  find  specific  ways  to  enhance  the  training 
effectiveness  and  efficiency  of  VE  systems.  Results  from  that  research  can  then  be  applied  to 
training  dismounted  Soldiers,  small  unit  teams,  and  small  unit  leaders.  The  results  should  also 
apply  to  other,  closely  related  simulation  environments  such  as  embedded  systems  and 
augmented  reality  systems  (discussed  below).  The  research  approach  must  address  VE-specific 
capabilities  that  can  enhance  learning  and  skill  levels  for  critical  or  important  dismounted  Soldier 
tasks.  This  line  of  research  has  become  more  important  because  the  Army  is  increasing  the 
emphasis  on  adaptive  and  flexible  training  in  response  to>new  equipment  (TRADOC  Pamphlet 
525-66,  2005)  and  the  current  operating  environment. 

Increased  Requirements  for  Improved  Training 

The  changes  in  equipment  and  training  needs  are  resulting  in  increasing  cognitive  loads 
being  imposed  on  the  squad  and  platoon  leaders.  Early  changes  were  the  result  of  strategic  plans 
developed  for  Soldiers  and  Soldiers  equipment  through  the  Land  Warrior  and  Future  Force 
Warrior  programs  (National  Research  Council,  1997,  TRADOC  Capability  Development 
Document  for  Ground  Soldier  System,  2004).  Other  changes  in  training  are  being  caused  by  the 
current  operating  environment  from  the  operations  in  Afghanistan  and  Iraq.  According  to  the 
National  Research  Council  (1997),  learning  to  execute  common  and  repetitive  skilled  activities 
within  the  three  most  significant  Infantry  missions  may  be  necessary  for  the  success  of  those 
missions.  As  the  information  load  of  small  unit  leaders  and  Soldiers  increases,  rapid  acquisition 
of  skill  in  dealing  with  that  information  load  will  be  even  more  important.  One  way  to  keep 
technology-based  increases  in  information  flow  within  the  cognitive  resource  bounds  of  the 
small  unit  leader  may  be  to  reduce  the  load  required  for  the  successful  performance  of  other 
ubiquitous  activities  through  improved  training. 


2 


Serendipitously,  one  way  to  conduct  research  on  the  instructional  tactics,  techniques,  and 
features  for  improved  training  within  this  technological  context  is  through  the  use  of  VE 
systems,  conceptually  referred  to  as  constructive  approaches.  Fielding  fully  immersive  VE- 
based  training  is  currently  constrained  by  many  technological  and  cost  factors.  Fully  immersive 
VE-based  instruction  (e.g.  using  helmet  mounted  displays  to  totally  control  the  visual  field)  will 
continue  to  be  relatively  expensive  and  difficult  to  deploy  for  individual  training  in  the 
immediate  future.  However  many  applications  can  and  will  use  less  than  fully  immersive 
systems  (for  example  game  systems,  Mayo,  Singer,  &  Kusumoto,  2005).  It  is  also  clear  that 
there  are  multiple  levels  of  virtuality  within  which  instructional  strategies  can  be  implemented. 
For  example,  there  is  a  continuum  of  artificiality  that  can  support  task  activities  (Milgram,  et  al., 
1994).  At  one  end  of  this  spectrum  are  VE  systems  in  which  everything  is  artificial,  and  at  the 
other  end  is  everyday,  normal  reality.  However  there  are  displays  that  can  add  stimuli  to  normal 
reality,  creating  an  augmented  reality  (AR),  as  well  as  physically  real  objects  (e.g.  weapons  or 
tools)  that  can  be  added  to  VEs.  This  range  has  been  labeled  Mixed  Reality  (Milgram,  et  al., 
1994)  which  includes  fully  immersive  VEs  and  the  lower  fidelity  applications  (for  example  those 
based  on  gaming  technology  and  presented  on  desktop  screens). 

AR  technology  adds  artificial  stimuli  (e.g.  sounds,  visual  displays)  to  the  stimuli  from  the 
real  world.  As  an  example,  a  heads-up  display  could  overlay  heat  signatures  to  the  real  world 
view.  VE-based  gaming  technology  can  also  introduce  artificial  stimuli  to  media  or  graphics 
representing  the  real  world,  and  use  the  entire  stimulus  set  for  interactive  training  (McCollum,  et 
al.,  2004;  Singer,  Comer,  &  Gross,  2004).  In  these  lower  fidelity,  less  immersive  systems, 
simulated  cues  can  also  be  added  to  the  natural -looking  stimulus  set  (the  artificially  presented 
environment).  For  example,  within  a  graphically  presented  virtual  world,  the  instructional 
system  may  insert  arrows  to  point  out  possible  locations  for  improvised  explosive  devices,  in 
order  to  train  environmental  inspection  techniques. 

So  the  argument  is  that  in  addition  to  knowing  the  real  world  stimuli  (e.g.  weapons  visual 
representations)  and  functionality  (e.g.  sound  of  firing,  effects  downrange)  that  have  to  be 
represented  for  training  in  a  simulation,  the  instructor  may  wish  to  insert  special  cues  for 
instructional  purposes.  Therefore,  instructional  design  research  into  instructional  tactics, 
techniques  and  features  may  provide  results  that  can  be  used  in  fully  immersive  systems,  AR- 
based  training  systems  (Kirkley,  et  al.,  2003.  2004),  game-based  simulation  systems  (McCollum, 
et  al.,  2004;  Singer,  Comer,  &  Gross,  2004),  or  in  any  class  of  mixed  reality  training  system. 

In  order  to  support  as  wide  a  range  of  applications  as  possible,  the  focus  in  this  effort  is  on 
initial  skill  training.  The  goal  of  skill-based  training  is  to  efficiently  move  someone  from  initial 
declarative  knowledge  (being  able  to  answer  questions  about  rules  or  concepts)  through  slow  and 
error-ridden  performance  to  faster,  more  competent,  and  less-effortful  performance.  In 
psychological  terms,  this  means  moving  the  learner  from  the  declarative  state  knowledge  of  a 
skill  into  some  level  of  procedural  knowledge,  and  within  the  procedural  area  from  more 
effortful  and  error-ridden  executive-controlled  processes  toward  more  rapid  and  correct 
performance  with  less  effort.  As  discussed  below,  procedural  knowledge  is  about  how  to  do 
things,  and  is  often  associated  with  performance  automaticity.  Automaticity  refers  to  processes 
that  are  easily  initiated  by  selected  classes  of  stimuli,  that  proceed  with  little  probability  of  error, 
and  require  minimal  cognitive  effort.  Initial  skill  training  sets  the  learner  on  the  path  toward 
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automaticity,  and  instructional  strategies  with  appropriately  used  instructional  features  should 
make  those  first  steps  as  efficient  and  effective  as  possible. 

The  approach  used  in  this  effort  was  to  identify  an  instructional  strategy  and  appropriate 
features  used  in  non-VE  domains  in  academia,  industry,  and  certain  military  training  programs 
(e.g.  see  Loube,  2001),  and  then  examine  their  potential  for  use  in  dismounted  Soldier  VE 
training  systems.  The  first  step  was  to  identify  a  group  of  common  small  group  leader  tasks, 
based  on  an  assessment  of  activities  necessary  for  US  Army  operations,  as  candidates  for  fully 
immersive  VE-based  training  with  instructional  features.  Second,  informed  by  empirical 
findings  and  theoretical  work,  we  defined  and  conceptually  integrated  instructional  goals, 
features,  strategies,  and  tactics  to  provide  a  reasonable  degree  of  order  and  clarity  to  our 
approach.  Third,  we  identified  several  instructional  features  within  an  instructional  strategy  that 
seemed  likely  to  improve  the  effectiveness  and  efficiency  of  VE-based  training.  Finally,  we 
conducted  an  experiment  that  tested  the  effect  of  two  instructional  features,  within  the  constraints 
of  a  commonly  used  instructional  strategy,  as  applied  to  small  group  leader  task  accomplishment. 
The  results  of  this  research  are  reviewed  in  order  to  suggest  guidelines  for  using  instructional 
features  in  future  VE  and  AR  training  programs.  These  results  and  the  information  from  related 
literature  are  also  used  to  suggest  further  research  on  the  complicated  and  interrelated  aspects  of 
Instructional  Strategies,  Tactics,  and  Features. 

Common  Soldier  Tasks 

Soldiers  use  a  large  number  of  specific  types  of  knowledge,  skills,  and  abilities  (KSAs)  in 
the  course  of  operations.  Jacobs  et  al.  (1994)  analyzed  tasks  and  activities  of  individual 
combatants  for  AR1  in  order  to  recommend  behavioral  and  technological  requirements  for  VE 
training  (for  examples  see  Table  1).  The  analysis  revealed  230  unique  activities  associated  with 
67  Soldier  tasks,  with  most  tasks  supported  by  multiple  activities.  For  instance,  the  task  of 
clearing  a  building  involves  giving  verbal  orders,  calling  in  preplanned  fire  requests,  hearing 
orders,  identifying  safe  and  danger  areas,  perceiving  the  relative  position  of  other  units,  and  at 
least  34  other  unique  activities.  In  addition,  Jacobs  et  al.  rated  each  activity  according  to  its 
primary  and  secondary  sensory  modality  (auditory,  visual,  tactile,  and  force  feedback),  as  well  as 
the  required  effector  — the  primary  method  by  which  the  response  of  the  Soldier  is  monitored 
and  injected  into  the  VE  simulation  (hand,  finger,  head,  body,  speech,  and  instrumented  objects). 
Jacobs  et  al.  also  rank  ordered  activities  according  to  the  frequency  of  the  activity  in  various 
tasks,  and  how  well  VE  technology  could  support  the  activity  in  training  systems.  Table  1  lists 
the  top  15  activities  and  ratings  for  sensory  modalities  and  effectors  from  the  Jacobs  et  al.  work. 

The  activities  identified  by  Jacobs  et  al.  (1994)  are  categorized  by  sensory  and 
psychomotor  functions,  yet  many  computer-based  training  scenarios  used  by  the  U.  S.  Army  to 
teach  command  decision  making  clearly  emphasize  higher-level  cognitive  functions.  For 
instance,  a  scenario  for  training  platoon  leaders  on  how  to  handle  Operations  Other  than  War 
(OOTW)  events  requires  trainees  to  assess  a  situation,  select  a  course  of  action,  and  properly 
implement  the  action.  Training  methods  are  used  to  provide  information  and  point  out  stimuli, 
and  in  general  help  guide  the  trainee  through  a  series  of  decision  points.  These,  and  other  current 
U.  S.  Army  training  programs,  support  the  concept  that  skills  such  as  problem  identification. 
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decision  making,  problem  solving,  and  action  implementation  deserve  more  attention  in  VE- 
based  training. 


Table  1: 


Top  Activities^  for  Soldier  Tasks  and  Supportability  by  VE 


Activity 

Sensory  Modality 

Primary  Secondary 

Effector 

1 .  Give  verbal  orders 

Sound 

Speech 

2.  Use  password 

Sound 

Speech 

3.  Blow  whistle  for  signal 

Sound 

Hand/Head 

4.  Call  in  preplanned  fire  requests 

Sound 

Speech 

5.  Inspect  for  correct  “Soldier’s  load” 

Visual 

Force  Feedback 

Hand/Head 

6.  Hear  orders 

Sound 

Speech 

7.  Operate  radio  or  telephone 

Sound 

Visual/Tactile 

Hand/Head/Instr.  Object 

8.  Identify  safe  or  danger  area 

Visual 

Head/Body/Hand 

9.  Perceive  relative  position  of  other  units 

Visual 

Sound 

Head/Body/Speech 

10.  Give  hand  and  arm  signals 

Visual 

Hand/Body 

1 1 .  Move  in  accordance  with  directions 

Visual 

Tactile/Force 

Feedback/Sound 

Hand/Head/Body 

12.  Visually  search  for  enemy 

Visual 

Head/Speech 

13.  Identify  hand  and  arm  signals 

Visual 

Head/Speech 

14.  Aim  and  fire  individual  weapon 

Visual 

Tactile/Sound 

Head/Body/Hand 

15.  Aim  and  fire  crew  served  weapon 

Visual 

Tactile/Sound 

Head/Body/Hand 

Note.  Adapted  from  “Training  Dismounted  Soldiers  in  Virtual  Environments:  Task  and 


Research  Requirements”  by  Jacobs  et  al.,  1994. 


We  are  limited,  however,  in  the  number  of  tasks  we  can  evaluate  in  order  to  show  how  the 
optimal  use  of  instructional  features  might  improve  VE-based  training.  For  our  research 
purposes,  we  abstracted  many  of  the  tasks  and  activities  from  Jacobs  et  al.  (1994),  as  well  as 
several  cognitive  tasks,  into  three,  higher-order  groups  of  tasks.  These  three  groups  are 
collectively  referred  to  as  Position/Maneuver  expertise  (PME)  and  are  integral  to  the  critical 
missions  identified  in  the  National  Research  Council  report  (1997).  The  three  groups  incorporate 
both  perceptual-psychomotor  and  cognitive  dimensions: 

1,  Enemy  Position:  refers  to  knowledge  and  awareness  of  enemy  locations  and  firepower 
capabilities  (type,  range,  threat  zone,  etc.).  Example  activities  from  Jacobs  et  al.’s  list  include: 

♦  Identify  safe  or  danger  area 

♦  Visually  search  for  enemy 

♦  Identify  areas  that  mask  supporting  enemy  fires 

♦  Perceive  relative  position  of  weapon  fire  (own  forces  and  opposing  forces) 

♦  Identify  obstacles 

♦  Identify  light  reflected  from  shiny  objects 

♦  Identify  glow  from  cigarette 

♦  Identify  flashes  from  enemy  weapons 

♦  Discern  direction  enemy  is  moving 

♦  Identify  enemy  Soldiers 
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2.  Own  Position:  refers  to  knowledge  and  awareness  of  trainee’s  fellow  Soldiers,  troops, 
and  other  assets  (e.g.  artillery,  tanks).  Examples  include: 

♦  Call  in  preplanned  fire  requests 

♦  Identify  safe  and  danger  area 

♦  Perceive  relative  positions  of  other  units 

♦  Move  in  accordance  with  directions 

♦  Identify  hand  and  arm  signals 

♦  Identify  support  position  to  fire  on  enemy 

♦  Maintain  position  relative  to  other  personnel 

♦  Identify  overwatch  position 

♦  Discern  location  within  an  area 

♦  Identify  firing  positions  in  urban  area 

3.  Access  Identification:  refers  to  knowledge  and  awareness  of  avenues  for  logistics 
support  (Medical  Evacuation,  supply  vehicles,  other  troops).  For  example,  evaluating  if  a 
helicopter  could  land  in  the  area  and  identifying  events  that  could  prevent  future  access  (building 
collapse  due  to  bombing,  enemy  movement  into  open  field,  etc.).  Examples  include: 

♦  Identify  safe  and  danger  area 

♦  Identify  obstacles 

♦  Discern  location  within  an  area 

♦  Count/inventory  expendable  supplies 

♦  Identify  approach  to  a  landing  zone  (LZ)/drop  zone  (DZ)  free  of  tall  trees,  etc. 

♦  Identify  distribution  points  for  supplies 

♦  Identify  LZ/DZ 

♦  Identify  slopes  which  must  be  climbed 

♦  Inspect  boat  landing 

♦  Mark  LZ/DZ 

For  the  most  part,  these  tasks  are  complex,  require  visual  and  auditory  stimuli  based  on 
terrain  and  environmental  effects,  and  involve  dynamic  interaction  with  other  personnel.  The 
development  of  expertise  in  these  areas  could  be  fostered  by  performing  the  tasks  within  varying 
and  efficiently-repeatable  scenarios  that  require  the  execution  of  these  complex  procedures.  VE- 
based  training  appears  to  be  well  suited  for  meeting  these  requirements. 

Task  Selection 

The  minimal  task  analysis  conducted  by  Jacobs  et  al.  (1994)  focused  on  the  use  of 
projected  VE  technology  that  could  be  used  in  the  relatively  near  future.  We  have  used  that 
information  and  judgments  as  a  basis  for  selecting  relatively  typical  activities  that  are  sufficiently 
and  quickly  trained  that  they  can  be  used  as  the  basis  for  simulation-based  training  research.  By 
reviewing  Field  Manual  7-8,  Infantry  Rifle  Platoon  and  Squad  (2001)  the  specific  information 
needed  for  instruction  and  the  development  of  a  VE  for  application  and  practice  of  the  task 
activities  was  derived.  Relatively  standard  task  analytic  approaches  for  simulation-based 
training  were  used  in  this  process  (Hays  &  Singer,  1989). 
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Two  knowledge  types  were  utilized  in  each  of  the  three  task  clusters  listed  above,  and  in 
the  tasks  selected  as  a  basis  for  research:  declarative  and  procedural  knowledge.  In  brief, 
declarative  knowledge  refers  to  knowledge  of  facts,  schemata  and  propositions,  the  what  of  the 
task,  whereas  procedural  knowledge  concerns  how  to  do  something.  This  requires  converting 
presented  information  into  particular  cognitive  instantiations  -  a  mental  model.  For  example,  a 
Soldier  learns  the  basic  procedures  of  securing  a  building,  such  as  search  patterns  and  movement 
protocols,  through  documents,  presentations,  and  instruction.  This  declarative  knowledge  is  then 
called  on  when  the  Soldier  gains  procedural  knowledge  by  actually  practicing  the  application  of 
structured  rules  for  searching  and  securing  a  building,  for  example  during  standard  field  training 
exercises.  In  this  manner,  the  declarative  knowledge  (the  describable  "what")  is  elaborated  into 
procedural  knowledge  (the  observable  "how"),  and  the  application  of  the  knowledge  is 
demonstrated  for  performance  measurement. 

Successful  Soldiers  need  to  make  the  transition  from  declarative  to  procedural  knowledge 
almost  immediately,  rapidly  decreasing  the  amount  of  thought  or  diversion  of  attentional 
resources  needed  for  correct  performance.  The  situationally  correct  instantaneous  application  of 
procedural  knowledge,  or  procedural  automaticity,  should  therefore  be  the  ultimate  goal  for  any 
procedural  training  program.  Soldiers  in  particular  must  be  able  to  collect  information  from  the 
environment,  assess  the  meaning  and  importance  of  the  information,  and  immediately  apply  their 
knowledge  to  choose  the  optimal  course  of  action. 

Unfortunately,  reaching  automaticity  is  a  challenge.  Although  extended  practice  is 
necessary  to  produce  automaticity  for  most  tasks  (Logan  &  Klapp,  1991;  Palmeri,  1997),  simply 
increasing  practice  duration  is  not  sufficient  (Fisk,  Ackerman,  &  Schneider,  1987).  For  practice 
to  be  effective,  stimuli  from  the  environment,  and  rules  governing  responses,  must  be 
consistently  mapped  to  a  response  (Fisk,  Hodge,  Lee,  &  Rogers,  1990;  Logan,  1988).  In  cases 
when  stimuli  or  the  application  of  procedures  vary  from  one  practice  session  to  the  next,  as 
occurs  in  many  simulations,  automaticity  will  develop  more  slowly,  if  at  all.  Still,  through 
repeated  application,  performance  accuracy  and/or  speed  should  show  improvement. 

Differences  in  performance  improvement  over  trials  should  reflect  differences  in  the 
instructional  approach,  which  is  the  basic  assumption  behind  classic  transfer  of  training 
investigations  (see  Hays  &  Singer,  1989  for  a  review  of  approaches). 

Instructional  Framework 

The  tasks  require  acquiring  information,  applying  rules,  and  convergent  problem  solving. 
The  development  of  expertise  with  these  outcomes  could  be  fostered  by  some  combination  of  the 
following  instructional  tactics  (Jonassen  &  Tessmer,  1996,  except  where  noted): 

1 .  Describe  required  performance,  conditions,  and  criteria 

2.  Determine  purposes  or  relevance  of  content 

3.  Provide  authentic,  functional  tasks 

4.  Present  facts,  concept/rul e/principle  definitions  and  attributes 

5.  Provide  prototypical  examples/demonstrations 

6.  Provide  worked  examples 

7.  Provide  graphic  cues;  lines,  colors,  boxes,  arrows,  highlighting  (Jonassen,  Grabinger, 

&  Harris,  1990) 
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8.  Provide  oral  cues:  oral  direction  (Jonassen,  Grabinger,  &  Harris,  1990) 

9.  Provide  special  information  in  windows  (Jonassen,  Grabinger,  &  Harris,  1990) 

1 0.  Describe  key  aspects  of  information 

1 1 .  Apply  in  real  world  or  simulated  situations  (near  transfer) 

12.  Change  contexts  or  circumstances  (far  transfer) 

13.  Provide  confirmatory,  knowledge  of  correct  response 

14.  Provide  corrective  and  remedial  feedback 

15.  Provide  expert  solution  models 

1 6.  Provide  hierarchical  prerequisite  sequence 

17.  Provide  procedural,  job  sequence 

18.  Demonstrate  strategy  corrections 

19.  Review  prerequisite  skills  or  knowledge 

20.  Create  images 

21 .  Create  story  lines:  narrative  descriptions  of  information 

We  believe  mission  rehearsal  in  a  relatively  complex  VE  fits  the  task  requirements 
outlined  above,  and  can  help  train  Soldiers  to  this  procedural  automaticity.  However,  given  the 
obstacles  to  achieving  automaticity,  doing  so  will  require  the  use  of  strategies  and  techniques 
that  optimize  the  learning  environment  to  ensure  that  procedural  knowledge  is  consistently 
mapped,  or  related  to,  appropriate  actions.  We  therefore  examined  relevant  literature  related  to 
instructional  strategies  and  features  used  to  support  such  strategies,  in  order  to  identify  several 
promising  approaches  that  could  be  used  in  future  VE-based  training  programs. 

Conceptual  Integration:  Instructional  Goals,  Strategies,  Tactics,  and  Features 

Finding  the  best  way  to  impart  declarative  and  procedural  knowledge  in  an  explicit  and 
directed  manner  has  been  a  major  research  focus  in  a  number  of  scientific  and  applied  domains, 
particularly  in  the  cognitive  psychology,  educational  psychology,  industrial  and  organizational 
(I/O)  psychology,  and  military  disciplines.  Educators,  for  instance,  have  continually  sought 
better  ways  to  teach  their  students,  from  improved  textbooks  and  lecture  formats  to  computer- 
based  simulations  of  basic  concepts  (e.g.,  Graham,  Alloway,  &  Krames,  1994;  Horton,  1994; 
Savelsbergh,  de  Jong,  &  Ferguson-Hessler,  2000).  Instructional  Systems  researchers,  in  the  end, 
are  striving  to  identify  “the  extent  to  which  various  arrangements  and  characteristics  of  stimuli 
promote  learning”  (Boldovici,  1 992,  p.2). 

One  purpose  of  this  report  is  to  bring  sufficient  order  to  the  instructional  literature,  as  it 
applies  to  VE-based  training,  to  enable  a  taxonomic  approach  to  be  developed  that  can  support 
research  planning  and  eventual  optimization  of  future  VE-based  training  systems.  To  this  end, 
we  have  attempted  to  define,  contrast,  and  place  in  a  hierarchical  relationship  the  terms 
instructional  goal,  strategy,  tactic,  and  feature.  Unfortunately,  the  abundance  of  research  on 
directed  learning  has  made  it  difficult  to  define  and  differentiate  between  these  concepts.  For 
example,  multiple  terms  are  often  used  to  describe  similar  ideas  or  concepts.  Nevertheless,  we 
argue  that  for  most  directed  learning  approaches,  there  are  generally  four  main  components: 
goals,  strategies,  tactics,  and  features. 
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Definitions 


Instructional  goals,  approaches,  strategies,  tactics,  and  features  relate  to  one  another  in 
structured  and  supportive  relationships.  First,  if  we  accept  that  directed  learning  is  the 
purposeful  transfer  of  information,  knowledge,  skills,  abilities,  and/or  attitudes  from  one  source 
(e.g.,  instructor,  computer  software,  simulation,  or  other  system)  to  an  individual  or  group  (Hays, 
2001),  then  the  purpose  of  a  directed  learning  program  can  be  termed  the  Instructional  Goal. 

This  purpose  has  also  been  referred  to  as  the  instructional  objective,  outcome,  or  task.  Second, 
directed  learning  programs  must  have  one  or  more  (usually  multiple)  explicit  approaches,  or 
Instructional  Strategies —  a  “plan,  method,  or  series  of  activities  aimed  at  obtaining  a  specific 
goal”  (Jonassen  &  Grabowski,  1993,  p.20).  This  leads  to  Instructional  Tactics  which  are 
“specific  actions  which  are  well-rehearsed  and  are  used  to  enable  the  strategy”  (Jonassen  & 
Grabowski,  1993,  p.20).  Instructional  tactics  are  maneuvers  or  manipulations  embedded  within  a 
strategy  that  can  be  used  to  change  a  learner’s  knowledge  state  so  that  it  enables  the  learner  to 
reach  the  instructional  goal.  The  tactics  should  enable  a  learner  to  integrate  or  absorb  or 
assimilate  new  knowledge  into  their  repertoire.  Finally,  Instructional  Features  are  “elements  of 
training  devices  that  can  improve  training  efficiency  on  individual  tasks”  (Sticha,  et  al.,  1990, 
p.17).  Instructional  Features  thus  refers  to  a  wide  variety  of  tools  and/or  techniques  which 
instructors  can  use  to  support  and  implement  the  instructional  tactics  and  strategies.  Based  on 
these  definitions  and  our  presented  relational  structure,  it  should  be  clear  that  these  concepts  can 
be  organized  hierarchically,  as  depicted  for  the  example  training  goal  in  Figure  1 . 


Instructional  Goal:  Identify  &  Select  Landing  Zone 

1 

Instructional  Strategy:  Use  Simulation  to  Improve  " 

Application  of  Landing  Zone  Area  Rules 

/ . 

>  Instructional  Strategy:  . 

.  'f  Instructional  Strategy: 

Provide  Adjunct  Cue 

Support  Cognitive 

t 

Elaboration 

Instructional  Tactic:  Use 

Graphics  to  Indicate 

Instructional  Tactic:  Use 

Important  Areas 

Auditory  (Oral)  Prompts 

Instructional  Feature: 

Instructional  Feature:  £  • 

Instructor  Selectable  Area 

Channel  enabling  ^ 

Addressed  by  Graphic 

InstructorfTrainee  f  F  / 

V' .  7 •  "  -  -  -  •  .  - 

Communication 

Figure  I.  Relationship  among  instructional  goal,  strategies,  tactics,  and  features. 
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In  Figure  1,  the  box  on  the  bottom  left  describes  an  instructional  feature:  a  tool  enabling 
the  instructor  to  select  an  area  that  is  important  for  the  application  of  Landing  Zone  rules.  For 
example,  the  instructor  may  want  to  indicate  that  a  nearby  tree  is  too  tall  for  a  helicopter  glide 
path.  The  instructional  feature  enables  the  instructional  strategy  of  providing  cues,  implemented 
through  the  instructional  tactic  of  using  graphics  cues  (an  arrow  in  our  research,  see  the  example, 
and  discussion  below)  to  point  out  critical  environmental  features.  This  enables  the  instructional 
strategy  of  providing  an  adjunct  cue  (something  not  normally  in  the  environment,  see  discussion 
below)  in  order  to  aid  in  the  correct  application  of  known  rules. 

The  second  Instructional  Strategy  (see  Figure  1)  was  to  provide  support  for  cognitive 
elaboration  in  the  application  of  Landing  Zone  rules.  The  enabling  tactic  was  to  provide  oral 
cues  (oral  questions)  that  elicited  the  most  applicable  rule  given  the  environmental  and  task 
situation.  The  tactic  of  providing  oral  cues  during  rehearsal  in  a  simulation  is  in  turn  supported 
and  enabled  by  an  instructional  feature  -  a  communications  channel  between  the  trainee  and 
instructor. 

Seldom  do  strategies,  tactics,  or  features  appear  in  sufficiently  clear  isolation  within  an 
instructional  program  that  would  enable  analysis  of  their  effects.  In  this  experiment  we  are 
providing  clear  and  isolated  cue  combinations  of  an  Instructional  Feature  Graphic  (directional 
arrow)  and  oral  question  tactic  as  cues  and  feedback  (coaching),  in  order  to  measure  the  benefit 
to  the  instructional  strategy.  It  should  be  noted  that  a  feature  can  support  more  than  one  strategy 
or  tactic.  For  example,  the  graphic,  when  used  to  highlight  an  error,  may  also  implement  a 
feedback  strategy  through  an  informational  feedback  tactic  (Bailey,  1 982). 

Instructional  Feature  Categories 

As  noted  above,  a  number  of  Instructional  Features  (IF)  can  be  used  to  support  and 
implement  specific  instructional  strategies.  These  features  can  be  categorized  along  four 
dimensions  based  on  characteristics  of  the  feature  and  how  it  is  applied:  Temporal, 
Adjunct/Augment,  Naturalness,  and  Control.  As  with  strategies,  these  categories  are  constructed 
to  be  of  most  relevance  and  use  when  addressing  simulations,  whether  for  training  or  skill 
maintenance.  The  categories  will  apply  to  all  training  approaches,  in  general  form,  but  make 
most  sense  for  simulation-based  training. 

The  Temporal  category  concerns  when  the  IF  is  implemented;  before,  during,  or  after  a 
training  task  or  event.  For  example,  many  demonstration  features  occur  prior  to  participants’ 
practicing  tasks  or  skills.  Review  features,  on  the  other  hand,  may  occur  at  any  time  surrounding 
an  event.  There  are  also  features  that  can  slow  or  speed  portions  of  practiced  tasks  or 
presentations,  skip  over  segments,  and  repeat  them  as  necessary  or  desired.  And  of  course,  as 
shown  in  Figure  1 ,  there  are  many  IF  applications  that  provide  support  during  training  strategies 
(e.g.  coaching). 

The  Adjunct/Augment  category  concerns  how  the  feature  is  implemented,  and  addresses  the 
stimuli  inherent  in  the  information  presentation  (Boldovici,  1992).  Practice  with  actual 
equipment,  simulations,  media  presentations,  etc.  will  (or  should)  present  the  necessary 
initiating,  guiding,  and  correct-performance  reinforcing  stimuli  for  every  required  activity.  Even 
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text  or  lecture  will  identify  these  stimuli  so  that  the  performer  can  correctly  attend  to  and  use 
them.  Augmenting  an  initiating,  guiding,  or  reinforcing  cue  means  manipulating  its  normal 
sensory  characteristics  to  make  it  more  noticeable  or  salient  (Boldovici,  1992).  Adjunct  cues  or 
stimuli  are  those  that  do  not  normally  belong  in  the  target  task  environment,  and  the  IF  consists 
of  inserting  something  new  which  calls  attention  to  or  makes  more  salient  the  important 
initiating,  guiding,  or  reinforcing  stimuli.  In  general,  this  category  relates  to  features 
implemented  during  performance  of  a  training  task. 

The  Control  category  concerns  who  is  responsible  for  administering  or  directing  the 
feature.  In  most  cases,  features  are  either  trainee  controlled,  instructor  controlled,  or  are 
automated  such  that  the  VE  or  computer-based  system  recognizes  events  and  then  presents  the 
feature. 

Finally,  the  Naturalistic  category  concerns  the  degree  to  which  the  feature  imitates  or  has 
the  appearance  of  stimuli  in  the  real  world  task.  Most  features  are  classified  as  unnatural 
because,  by  their  very  nature,  the  features  add  to  or  alter  the  natural  environment  in  order  to 
train. 


As  minimally  discussed  above,  practice  alone  is  unlikely  to  provide  or  impart  automaticity 
in  knowledge  application,  for  example  in  performing  a  task.  Several  researchers  have  argued 
(Fisk  et  al.,  1990;  Logan,  1988)  that  in  a  training  session,  stimuli  must  be  consistently  paired 
with  appropriate  responses  if  automaticity  in  the  form  of  accurate  and  correct  performance  is  to 
develop  at  a  relatively  rapid  pace.  Unfortunately,  this  leads  to  a  dilemma  because  training 
Soldiers  repeatedly  with  the  exact  same  stimuli — the  approach  that  most  quickly  builds 
automaticity — could  lead  to  inflexibility.  In  other  words,  Soldiers  might  leam  how  to  rapidly 
and  automatically  respond  to  identical  conditions  in  real  situations,  but  may  not  be  able  to 
rapidly  adapt  and  transfer  this  performance  from  one  situation  to  another,  as  is  the  general  goal 
in  Army  training. 

The  Army  does  not  need  to  train  Soldiers  all  the  way  to  automaticity,  but  Soldiers  do  need 
to  reach  a  well-practiced,  easily  performed  and  highly  skilled  level,  with  skills  that  are  readily 
adaptable  and  transferable.  The  appropriate  knowledge  should  be  rapidly  and  easily  retrieved, 
and  the  important  decisions  should  be  quickly  and  easily  made  with  a  minimum  of  cognitive 
load.  The  application  of  selected  IFs  within  VE  simulations  may  be  a  promising  solution.  A 
major  advantage  of  VE-based  training  is  the  flexibility  of  inserting  stimuli.  Environments  can  be 
automatically  constructed  from  geographical  databases  and  satellite  imagery.  With  minimal 
alterations,  instructors  can  add  new  attributes  or  dimensions  to  VE  simulations  to  significantly 
change  the  trainees’  experience.  For  example,  the  ability  to  shift  from  daytime  to  nighttime 
operations,  add  night- vision  displays,  or  to  increase  the  number  of  opposing  forces  or  change 
their  strategies  and  actions,  can  be  preprogrammed  into  the  system  to  give  instructors  greater 
selection  and  control  of  environments  and  conditions.  This  flexibility  also  means  that  instructors 
can  employ  a  number  of  IF,  when  necessary,  to  enhance  training. 

The  first  step  to  providing  recommendations  about  IFs  for  use  within  Army  VE 
simulations  is  to  identify  features  that  have  shown  promise  in  improving  training  in  similar 
simulations  for  similar  tasks.  We  searched  the  literature  from  academia,  industry,  the  military, 
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and  similar  areas  for  studies  that  either  evaluated  or  employed  IFs  in  an  instructional  simulation. 
Although  a  search  using  reasonable  keywords  yielded  a  large  amount  of  material,  most  articles 
offered  little  concrete  information  about  the  efficacy  of  a  specific  IF,  primarily  because  the  goals 
were  not  centered  on  features  (i.e.,  a  feature  was  used,  but  no  evaluation  of  specific 
effectiveness).  Primarily  based  on  older  reviews  of  the  area  (Ayres  et  al.  1984;  Knerr  et  al., 
1998;  Sticha  et  ah,  1990),  in  addition  to  insights  from  discussions  with  our  developers  at  the 
Institute  for  Simulation  and  Training,  we  developed  a  set  of  three  IFs  that  appear  to  have  the 
greatest  potential  for  success,  and  therefore  deserve  empirical  evaluation. 

To  describe  the  selected  features,  we  used  one  of  the  three  common  task  groups  previously 
mentioned,  those  related  to  Enemy  Position,  as  a  framing  example.  In  addition,  each  feature  is 
related  to  one  or  more  instructional  strategies. 

Dynamic  Diagrams 

Computer-generated  diagrams,  graphics,  or  maps  that  demonstrate  tasks  or  procedures 
through  the  use  of  time-linked  elements  that  indicate  movement  and  sequence  are  referred  to  as 
dynamic.  For  example,  Park  and  Gittleman  (1995)  compared  Dynamic  Visual  Displays  with 
Static  Visual  Displays  in  their  effect  on  the  formation  of  mental  models  of  electronic  circuits. 
Although  the  authors  found  no  difference  on  mental  model  formation,  students  who  worked  with 
Dynamic  Visual  Displays  showed  better  learning  early  on  and  faster  overall  problem  solving. 


Figure  2.  Example  dynamic  diagram. 
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Figure  2  illustrates  a  dynamic  diagram  (as  well  as  can  be  done  given  print  media)  based  on 
a  previous  ARI  developed  VE  research  simulation  database  (Singer  et  al.,  2001).  The  line 
starting  in  the  small  vestibule  at  the  bottom  of  the  figure  extending  through  the  first  room,  in  and 
out  of  the  small  room  on  the  lower-right,  and  ending  in  the  second  room  on  the  right  illustrates 
the  correct  order  of  room  search  (i.e.,  the  “right  hand  rule”).  Trainees  would  see  this  line  appear 
and  then  extend  from  room  to  room  in  a  dynamic  path  illustrating  the  rule.  (Note  that  this 
dynamic  diagram  was  not  actually  used  in  that  research.) 

Strategies.  Dynamic  diagrams  could  implement  a  provide  examples  instructional  strategy 
through  a  demonstration  instructional  tactic  to  teach  or  further  explain  concepts.  The 
instructional  tactic  would  be  to  use  the  dynamic  diagram  instructional  feature  to  show  the 
performance  of  critical  situational  procedures.  For  Enemy  Position  tasks,  a  dynamic  diagram 
instructional  feature  could  be  used  to  show  known  enemy  movements,  artillery  ranges,  etc. 
Dynamic  diagrams  could  also  be  used  to  implement  a  provide  feedback  strategy  through  an 
information  feedback  tactic,  allowing  trainees  to  see  a  representation  of  themselves  performing 
the  procedures  in  question.  Dynamic  diagrams  like  this  have  been  implemented  in  an 
experimental  AAR  system  for  VE  simulations  (Knerr,  et  al.,  2002),  in  which  participants  can 
observe  their  own  and  others  movements  represented  as  icons  or  avatars,  leaving  colored  trails  or 
lines  indicating  the  traversed  path. 

Potential  Problems.  This  kind  of  feature  presents  two  developmental  difficulties;  display 
work  load  and  application  dynamics.  While  the  example  presented  above  is  relatively  simple 
and  direct,  implementing  a  dynamic  visual  display  is  not  trivial  and  can  add  significantly  to  the 
cost  of  display  development  if  the  illustration  becomes  complex.  For  example,  the  dynamics 
require  not  just  visually  modeling  all  components  of  a  piece  of  equipment  that  has  to  be 
disassembled,  but  also  showing  the  dynamic  motions  required  for  that  disassembly.  The  second 
issue  is  application  dynamics,  in  which  the  determination  of  important  aspects  of  a  situation,  how 
much  detail,  and  when  to  show  those  details  are  just  the  grossest  considerations.  A  rational  and 
effective  approach  to  evaluating  this  class  of  IF  would  be  to  address  a  single  generic  form  of  this 
feature,  a  prototypical  example  of  the  feature  developed  for  a  clearly  defined  category  of  skills, 
knowledges,  or  abilities. 

Text-based  Messages 

Text-based  messages  are  usually  text  windows  that  pop-up  into  the  field  of  view  (FOV)  at 
an  instructionally  appropriate  time  (see  Figure  3).  Common  behavioral  tenets  indicate  that  error 
messages  should  be  presented  immediately  after  the  trainee  makes  an  error,  although  they  can 
appear  during  an  activity.  Similar  approaches  have  proven  successful  in  training  pilots  in  flight 
simulators  (Skitka,  Mosier,  &  Burdick,  1999),  mission  control  personnel  on  satellite  deployment 
procedures  (Loftin,  Wang,  Baffes,  &  Hua,  1992),  and  showing  users  how  to  operate  the  Navy’s 
consolidated  Area  Telephone  System  (Cowen,  1993). 

Strategies.  Error  messages  could  afford  a  provide  feedback  strategy  through  a  provide 
corrective  and  remedial  feedback  tactic  (Jonassen  &  Tessmer,  1996)  as  in  Figure  3.  For  the 
Enemy  Position  class  of  tasks  and  activities,  these  pop-up  messages  might  provide  warnings, 
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emphasize  inspection  or  movement  protocols,  or  provide  reminders  for  the  types  of  cues 
indicating  enemy  movement  or  positions. 


Figure  3.  Example  text-based  message. 


Potential  problems.  The  crutch  effect  (Boldovici,  1992)  occurs  when  an  inserted  or  added 
feature  or  function  in  a  training  simulation  produces  a  reliance  on  that  feature  as  a  cue  for 
performance.  In  this  case,  the  effect  might  be  based  on  an  over-reliance  on  the  text  error 
feedback  that  transfers  to  real  world  situations  and  degrades  initial  transfer  performance.  As 
always,  implementation  issues  (e.g.,  form,  location,  timing,  duration)  can  present  problems  for 
the  addition  of  non-normal  cues  or  information  in  a  simulation.  In  this  case,  message  position  in 
the  FOV  remains  a  question.  If  appearance  is  dynamic,  based  on  performance  monitoring,  the 
text-window  potentially  could  overlay  important  cues  for  correct  performance  when  it  appears. 
At  this  point,  placing  the  message  toward  the  bottom  of  the  trainee’s  FOV,  such  that  around  75% 
of  normal  view  remains,  seems  to  avoid  most  of  the  apparent  location  problems  (see  Figure  3). 

Augmenting  and  Adjuncting  Cues 

These  categories  are  derived  from  a  theoretical  approach  developed  by  Boldovici  (1992)  in 
an  attempt  to  clarify  actions  that  can  be  taken  to  increase  the  meaningfulness  (salience,  to  use  his 
term)  of  stimuli  during  training.  Augmenting  means  emphasizing  or  de-emphasizing  the 
naturally  occurring  attributes  or  characteristics  of  the  task-important  initiating,  guiding,  or 
performance  feedback  cues  in  a  simulation  environment  in  order  to  alert  or  orient  trainees  to 
critical  stimuli.  This  can  be  accomplished  by  making  one  or  more  characteristics  of  existing 
critical  stimuli  more  salient  to  the  trainee.  For  example,  changing  the  form  or  duration  of  a 
natural  stimulus  by  enhancing  the  color  or  sound  would  be  augmenting  that  dimension  of  the 
stimulus.  Alternatively,  one  could  diminish  or  de-emphasize  the  surrounding  visual  stimuli  or 
sounds  so  as  to  make  the  key  characteristic  more  noticeable  (the  decrease  in  distracters  is 
described  as  fading  by  Boldovici).  Adjuncting  refers  to  adding  a  cue  or  stimulus  to  the 
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environment  that  does  not  normally  occur  (referred  to  as  supplementing  by  Boldovici). 
Employing  an  unnatural  stimulus  (e.g.,  a  colored  outline,  flashing  dots,  pointing  arrows,  etc)  will 
draw  attention  to  those  environmental  stimuli  (Boldovici).  This  approach  has  shown  limited 
promise  in  teaching  firefighters  how  to  recognize  the  direction  and  spread  of  brushfires 
(Lewandowsky,  Dunn,  Kirsner,  &  Randell,  1997). 

As  noted  by  Boldovici  (1992),  salience  or  meaningfulness  is  not  a  property  of  the 
environmental  stimuli  -  it  is  a  property  assigned  by  the  organism  during  perception.  Boldovici 
presents  a  somewhat  behavioristic  approach  to  stimuli  and  responses  in  a  task  environment, 
although  it  seems  more  reasonable  to  include  those  behavioral  tenets  within  a  more  cognitive 
framework.  To  that  end  we  can  restate  his  point  about  salience  in  the  following  way,  that  the 
trainee/leamer  assigned  meaningfulness  of  the  stimuli  is  the  key  to  a  cue’s  effectiveness  in  the 
learning  environment.  While  this  meaningfulness  can  be  derived  without  symbolic 
representation  or  cognitive  modeling  by  an  organism  through  behavioral  conditioning 
techniques,  transmission  of  information  through  language  can  also  lead  to  information  being 
incorporated  in  a  mental  model.  Therefore,  explaining  the  use  of  an  augmenting  cue  or  adjunct 
cue  to  the  learner  provides  the  basis  for  the  meaningfulness  of  important  stimuli  in  a  simulated 
task  environment.  If  a  normally  occurring  visual  or  auditory  characteristic  is  altered,  and  the 
trainee  does  not  perceive  the  change,  or  does  not  connect  the  perceived  change  to  important 
information  about  task  performance,  then  the  manipulation  is  without  value,  and  may  even  be 
detrimental  to  performance. 

For  example,  to  train  proper  building  search  movement,  “flashing”  doors  in  a  VE  building 
interior  could  be  used  to  indicate  the  room  that  team  members  should  enter  next,  as  illustrated  by 
the  surrounding  lines  around  a  door  in  Figure  4.  There  has  to  be  precursor  training  in  which 
“flashing”  has  been  clearly  explained  as  an  aiding  stimulus  for  where  to  go  next,  and  the  team 
members  have  to  remember  that  explanation. 


Strategies.  A  highlighting  tactic  would  implement  a  provide  cueing  system  strategy.  This 
tactic  would  be  appropriate  for  acquiring  bodies  of  information,  structuring  mental  models, 
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forming  concepts,  using  procedures,  and  convergent  problem  solving  (Jonassen  &  Tessmer, 

1996;  Park  &  Gittleman,  1996). 

Potential  problems.  There  does  not  appear  to  be  any  empirical  support  for  the 
effectiveness  of  this  approach  in  conjunction  with  movement  or  search  tasks.  However,  to  the 
extent  that  movement  and  search  are  based  on  reasoning  or  problem  solving  the  previous 
research  may  indicate  potential  for  increases  in  training  effectiveness. 

Instructional  Feature  Selection 

The  overriding  goal  of  the  present  research  was  to  identify  and  then  test  an  IF  for  use  in 
future  US  Army  VE  training  systems.  The  first  step  in  our  program  was  the  development  of  IFs 
for  evaluation.  As  the  features  were  developed  within  a  framework  of  training  task  and 
instructional  strategies,  the  opportunity  to  document  and  evaluate  the  development  easily  arises. 
Based  on  the  preceding  discussion,  we  chose  to  first  evaluate  an  adjunct-type  IF,  illustrated  in 
Figure  5,  whereby  a  position-linked  (both  with  the  trainee  and  the  location  of  importance)  arrow 
is  used  to  point  out  the  location  of  a  set  of  stimuli  that  must  be  re-evaluated  for  improved  task 
performance.  For  example,  the  participant  may  have  placed  or  marked  a  position  (see  discussion 
of  tasks,  below)  incorrectly,  rather  than  correctly  applying  the  rules  for  placing  or  marking  a 
position  (the  task  at  this  point  is  irrelevant).  When  the  instructor  detects  and  confirms  the  error, 
the  Attention-Direction  (AD)  Arrow  IF  can  be  used  to  point  to  the  incorrect  position,  or  to 
indicate  the  stimuli  that  should  be  used  to  determine  correct  placement  or  marking,  or  can  simply 
be  used  to  indicate  where  the  mark  should  have  been  placed.  The  arrow  is  designed  to  move  as 


Figure  5.  Position-linked  attention-direction  arrow  instructional  feature. 
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the  user’s  FOV  changes  so  that  it  always  points  at  the  location  specified  by  the  trainer/instructor. 
(Note  that  this  will  sometimes  mean  that  the  arrow  will  be  pointing  directly  at  the  trainee,  as  the 
indicated  location  is  directly  behind  the  trainee.) 

Using  the  classification  structure  introduced  above,  this  feature  is  categorized  as  an 
unnatural  IF,  one  that  trainers/instructors  manipulate  to  point  out  critical  areas  the  participant 
may  have  missed  in  the  visual  scan  of  the  environment.  Temporally,  the  arrow  feature  is 
employed  after  a  trainee  makes  an  error,  such  as  not  looking  at  potential  OpFor  positions,  and  is 
obviously  time-linked.  Categorization  of  the  arrow  feature,  in  the  context  of  IF  and  strategies,  is 
presented  in  Table  2. 

The  second  IF  chosen  for  this  research  was  Coaching.  The  coaching  was  presented  in  the 
form  of  an  oral  interrogative  spoken  by  the  experimenter  (heard  by  the  trainee  in  the  VE  system 
headset)  after  commission  of  an  error.  The  purpose  of  the  coaching  question  was  to  encourage 
participants  to  reconsider  previous  actions  for  appropriateness  based  upon  rules  that  had  been 
learned  during  the  introductory  training.  The  coaching  questions  used  were  designed  to  elicit  the 
correct  task-situational  application  of  one  of  the  rules  that  were  trained  before  the  mission 
planning  sessions  began.  This  was  accomplished  by  literally  asking  for  the  appropriate  rule, 
implicitly  providing  feedback  that  some  aspect  of  the  immediately  preceding  performance  or 
decision  was  incorrect,  and  explicitly  identifying  which  aspect  by  the  content  of  the  question 
stem.  If  participants  responded  incorrectly,  the  correct  rule  of  information  was  provided. 


Table  2. 


Categorization  of  a  Directional  Attention  Arrow  Instructional  Feature 


Instructional 

Strategies 

Temporal 

Adjunct/Augment 

Control 

Naturalistic 

Demonstration 

During 

Presentations 

Adjunct  Cue 

Pre-Planned  by 
Instructor 

Unnatural 

Practice 

NA 

NA 

NA 

NA 

Error 

Identify/Avoid 

After  failure  to 
look  at  area 

Adjunct  Cue 

Instructor 

Controlled 

Unnatural 

Review 

During 

Presentation 

Adjunct  Cue 

Instructor 

Controlled 

Unnatural 

The  timing  of  the  two  IF  interventions  was  an  important  consideration  for  us,  in  order  to  be 
able  to  cleanly  categorize  the  intervention  characteristics.  One  of  the  most  ubiquitous 
instructional  strategies  used  in  military  training  is  to  coach  during  the  performance  of  early  skill 
acquisition  trials.  Given  the  increasing  use  of  artificial  intelligence  methods  for  supporting 
training  strategies,  we  decided  to  mimic  Artificial  Intelligence  routines  by  designing  the  AD 
Arrow  and  Coaching  IFs  to  be  performance-error  linked  and  guided-discovery  based. 
Performance-error  linked  AD  Arrow  or  Coaching  intervention  means  that  no  adjunct  stimuli  or 
coaching  are  provided  until  a  decision  point  or  performance  step  is  reached  and  the  student  has 
performed  some  sub-task  segment  incorrectly.  During  the  training,  the  coach  then  immediately 
uses  a  context  sensitive  question  probe  to  elicit  from  the  trainee  the  known  rule  or  information 
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needed  for  a  correct  decision  or  performance  step.  Only  when  rules  are  not  correctly  retrieved  or 
expressed  by  the  trainee  does  the  “coach”  provide  direct  information,  in  the  form  of  pertinent 
information  or  an  important.  The  AD  Arrow  is  applied  in  the  same  fashion,  appearing  and 
indicating  a  location  of  importance  in  the  immediately  preceding  task  or  performance  step. 

Instructional  Feature  Experiment 

This  experiment  was  designed  to  investigate  the  training  effect  of  two  instructional  factors: 
Interrogative  Coaching  (IC)  intervention  and  an  Attention-Directing  (AD)  Arrow.  In  the 
experiment,  during  instruction,  a  participant  would  receive  an  oral  question  as  an  intervention,  an 
AD  Arrow  IF,  both,  or  neither.  It  also  produced  one  awkward,  but  necessary,  instructional 
situation  and  one  more  normal  situation:  the  awkward  situation  had  an  instructional  feature  that 
appeared  only  when  performance  errors  were  committed  and  had  no  accompanying  explanation, 
the  second,  more  normal  situation,  provided  coaching  when  an  error  had  been  made  but  no 
indication  was  given  of  the  location  of  important  cues.  As  initial  acquisition  of  skills  over 
repeated  trials  is  the  framing  context,  the  experimental  instruction  was  conducted  over  three 
repeated  sessions,  during  which  participants  practiced  and  received  feedback,  and  tested  during  a 
final  (fourth)  mission  without  instructional  interventions. 

The  application  of  these  conditions  results  in  a  standard  2  (presence  or  absence  of  coaching 
intervention)  x  2  (use  or  non-use  of  Attention-Direction  Arrow)  x  4  trial  repeated  measures 
experimental  design  (see  Figure  6).  The  general  hypotheses  were  that  some  coaching  is  better 
than  none,  and  that  attention-direction  information  will  have  positive  effects  over  no  attention- 
direction  information.  These  effects  were  measured  in  terms  of  accuracy  and  time  of 
performance  during  the  several  tasks  addressed  (the  dependent  variables  are  presented  in  greater 
detail  in  the  procedures  section,  below). 

The  general  hypothesized  differences  were  anticipated  to  be  the  case  both  over  the 
repeated  training  missions  (during  learning)  and  during  the  final  or  test  mission.  We  also  were 
interested  in  the  possibility  of  interactions  being  found  between  coaching  and  attention-direction 
IFs,  in  that  the  application  of  these  tactics  together  logically  should  be  better  than  either  applied 
separately.  However,  there  was  no  research  that  we  were  aware  of  that  would  support  prediction 
of  that  result.  It  was  also  expected  that  participants  would  improve  over  repeated  trials,  but  the 
most  interesting  outcomes  involved  the  effects  of  the  instructional  manipulations  over  the 
repeated  training  trials  and  the  final  outcome  mission. 
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AD  Arrow 


AD  Arrow 
Applied 


No  1C 

1C  Applied 

19t  Mission 


2nd  Mission 


3rd  Mission 


4th  or  Test 
Mission 


Figure  6.  Experimental  design. 


Method 


Participants 

Thirty-two  participants  were  recruited  from  the  University  of  Central  Florida  student 
population,  and  paid  or  given  class  credit  for  completing  participation.  Nine  of  the  participants 
were  female  and  twenty-three  were  male.  The  participants’  average  age  was  20.59  years. 

Thirty-one  of  the  thirty-two  participants  rated  themselves  as  average  to  highly  confident  on 
computers.  Thirty  of  the  thirty-two  rated  themselves  as  average  to  highly  skilled  on  video 
games.  While  ten  claimed  to  play  video  games  only  an  hour  or  less  per  week,  nineteen  played 
from  one  to  eighteen  hours  per  week,  with  three  claiming  twenty  plus  hours  per  week  (one 
claimed  the  maximum  of  thirty-five).  Fifteen  of  the  participants  had  never  experienced  virtual 
reality,  six  had  experienced  it  once,  and  only  one  claimed  more  than  ten  experiences.  All 
claimed  normal  vision,  with  only  one  reporting  any  color  blindness.  No  relationships  between 
these  variables  and  the  experimental  factors  were  found. 

Materials  and  Apparatus 

The  VE  scenarios  were  rendered  on  a  computer-based  system  consisting  of  linked  PCs. 
MotionStartm  sensors  were  used  to  track  participants’  physical  movements,  and  Virtual  Research 
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V8,m  head  mounted  displays  (HMD)  presented  head-slaved,  computer-generated,  stereoscopic 
color  imagery  to  the  participants.  Stereo  sound  for  object  collision,  gunshot  noises  (for  a  paint 
gun),  and  oral  coaching  were  provided  through  earphones  attached  to  the  HMD.  The  software 
was  written  by  1ST  using  Performer,  C++,  and  Java.  In  all,  the  experiment  required  the  use  of 
six  networked  personal  computers  and  one  stand-alone  personal  computer  for  questionnaire 
administration.  All  activities  took  place  within  a  safety  frame  referred  to  as  a  safety  pod.  The 
safety  pod  was  a  raised  platform  with  approximately  waist-high  encircling  railings  to  prevent 
falls  and  limit  movement.  The  HMD  with  head  tracking  sensor  and  body  position  sensors 
(including  shoulder  position,  hand  unit,  elbow,  and  ankle  sensors)  were  connected  to  a  backpack 
frame,  with  cables  running  up  to  a  boom  positioned  overhead.  This  arrangement  prevented 
excessive  movement,  precluded  falls,  and  allowed  relatively  free  movement  through  several  360 
degree  turns. 

Virtual  Environments.  Two  real-world  environments  were  digitally  modeled:  the 
Shughart-Gordon  Military  Operations  in  Urban  Terrain  (MOUT)  site  at  Ft.  Polk,  Louisiana,  and 
the  McKenna  MOUT  site  at  Ft.  Benning,  Georgia.  Each  of  these  environments  was  slightly 
altered  for  experimental  use,  and  had  two  different  starting  points  established.  This  allowed  each 
environment  to  be  used  twice  in  the  four-scenario  experimental  series. 

The  AD  Arrow  IF,  designed  by  1ST,  was  controlled  by  the  experimenter.  After 
participants  committed  an  error,  the  experimenter  would  select  the  location  by  using  a  mouse- 
control  key  combination,  causing  the  AD  Arrow  IF  to  appear  in  the  bottom  center  of  the 
participant’s  FOV,  with  a  changing  orientation  that  always  pointed  toward  the  Experimenter 
designated  target  location.  The  arrow  remained  in  the  FOV  until  the  participant  shifted  the 
center  of  their  gaze  to  within  a  20-degree  deviation  from  the  target  location,  whereupon  it  then 
disappeared  from  view.  This  center  of  view  determination  was  based  on  calculations  on  the 
FOV  in  the  HMD,  and  implemented  based  on  the  presumption  that  the  participant  would  actually 
observe  the  indicated  area  when  viewing  the  area  within  the  20-degree  deviation.  The 
interrogative  coaching  statements  were  scripted  for  critical  types  of  errors  and  read  over  the  VE 
sound  system  by  the  experimenter  at  the  appropriate  time,  based  upon  the  error  detected  by  the 
experimenter. 

Questionnaires.  Several  questionnaires  were  administered  before,  during,  and  after  the 
sequence  of  VE  scenarios.  All  questionnaires  were  implemented  in  an  Access1"1  database  for 
ease  of  administration  and  analysis.  The  Biographical  questionnaire  asked  questions  about 
participant  characteristics  and  experience,  primarily  with  video  games  and  VEs  (see  Appendix 
A).  The  Simulator  Sickness  Questionnaire  (SSQ;  Kennedy,  Lane,  Berbaum,  &  Lilienthal,  1993) 
was  used  to  address  the  level  of  symptoms  before,  during,  and  after  the  VE  episodes  (1  training 
experience  and  4  mission  sessions).  This  allowed  continuous  monitoring  of  the  participant’s 
physical  health  (see  Procedures,  below).  Other  questionnaires  included  the  Immersive 
Tendencies  Questionnaire  (ITQ;  Witmer  &  Singer,  1998),  administered  at  the  beginning  of  the 
experiment,  and  the  Presence  Questionnaire  (PQ  Vs.  3.0;  Witmer  &  Singer,  1998),  administered 
three  times  (see  Procedures,  below).  The  ITQ  addresses  the  participant’s  tendencies  toward 
immersion  and  involvement  in  general,  while  the  PQ  addresses  the  participant’s  level  of  self- 
perceived  immersion  and  involvement  in  the  immediately  preceding  virtual  reality-based 
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experience  or  episode.  These  data  are  collected  as  a  part  of  ongoing  ARI  SSRU  research  into 
perceptions  and  attention  during  VE  experiences. 

Procedures 

As  is  required  for  the  ethical  treatment  of  experimental  participants,  the  purposes  of  the 
research  were  explained  to  each  participant,  their  questions  were  answered  to  their  satisfaction, 
and  their  agreement  to  participate  in  the  research  was  recorded  (Ethical  Principles  of 
Psychologists  and  Code  of  Conduct,  American  Psychologist,  1992,  as  reprinted  in  Appendix  C  of 
the  Publication  Manual  of  the  American  Psychological  Association,  2001;  Army  Research 
Institute  for  the  Behavioral  &  Social  Sciences,  2003).  The  experimental  protocol  sequence  was 
as  follows: 

1 .  Upon  arrival,  each  participant  was  informed  about  compensation,  right  to  withdraw, 
and  basic  procedures  to  be  used  in  the  research. 

2.  A  demonstration  of  a  typical  planning  exercise  was  provided  on  a  large  flat  screen 
monitor.  A  different  VE  terrain  from  those  used  for  the  experimental  sessions  was 
used  for  this  demonstration,  to  prevent  environment  knowledge  transfer. 

3.  The  sequence  of  experimental  activities  was  reviewed  with  the  participant,  and 
agreement  to  participate  was  documented. 

4.  A  biographical  questionnaire  was  administered,  followed  by  the  ITQ,  and  the  first 
administration  of  the  SSQ. 

5.  The  participant  was  introduced  to  the  VE  experience  by  learning  to  move,  walk,  and 
pace  distances  in  the  environment,  as  well  as  using  the  marking  unit  (a  virtual  paint 
gun)  to  mark  positions  for  the  different  tasks  using  different  colors.  Participants  were 
coached  and  practiced  until  minimal  performance  standards  were  met.  The  training 
session  was  limited  to  twelve  minutes,  with  all  participants  successfully  completing 
training  in  that  time.  (It  should  be  noted  that  distance  examples  were  provided 
visually,  and  with  movement  time  examples.  No  accuracy  in  judging  distances  was 
taught  or  tested.) 

6.  Following  the  initial  VE  training  a  post-experience  SSQ  was  administered.  Criteria  for 
suspension  of  the  experiment  based  on  sickness  are  discussed  below.  After  completion 
of  the  SSQ,  the  first  PQ  was  administered. 

7.  A  manual  was  given  to  the  participants  (see  Appendix  B),  containing  instructions  and 
rules  for  the  dismounted  Soldier  synthetic  tasks:  a)  Bounding  Overwatch  (BO) 
movement,  b)  Landing  Zone  (LZ)  selection,  and  c)  Observation/Fire  Post  (OFP) 
emplacement.  A  questionnaire  and  map  test  (Appendix  C)  was  used  to  insure  that  the 
participant  understood  all  task  rules  and  requirements.  Participants  were  tested  on 
correct  knowledge  of  features  and  application  of  rules  using  a  multiple  choice  test 
(requiring  13  of  15  responses  correct)  and  a  short  map  planning  exercise  (no  more  than 
2  errors,  verbally  corrected  by  the  experimenter),  before  being  allowed  to  participate  in 
the  VE  exercises. 

8.  Following  successful  training,  participants  began  the  4  mission  experiment  cycle.  (The 
order  of  presentation  of  environments  was  counterbalanced.)  Each  mission  cycle  was 
conducted  as  follows: 
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•  Participants  were  first  reminded  of  the  mission  goals  and  criteria.  They  were  then 
given  a  mission  map  (see  Appendix  D)  and  asked  to  plan  their  movement  using 
BO  patterns  and  select  a  goal  location  for  a  helicopter  LZ  and  attendant  OFP 
emplacement. 

•  After  completing  a  pre-exercise  SSQ,  and  a  review  of  their  completed  map,  each 
participant  entered  the  appropriate  environment  and  conducted  a  mission 
reconnaissance  exercise  in  which  they  marked  their  planned  route,  LZ,  and  OFPs. 
Participants  were  reminded  before  beginning  that  they  could  change  the  map- 
planned  route  and  locations  based  on  environmental  appraisal  using  rules  from  the 
training  manual. 

•  Following  the  VE  mission  reconnaissance  exercises  (limited  to  twelve  minutes  in 
the  VE),  participants  removed  the  VE  equipment,  exited  the  safety  pod,  and 
moved  to  a  computer  to  complete  a  post-exercise  SSQ.  Immediately  after  the 
SSQ,  an  After  Action  Review  (AAR)  of  five  minutes  duration  was  administered. 
During  the  AAR  only  previously  uncorrected  errors  were  called  out,  and  as  with 
the  1C  Strategy  Condition  rules,  the  participant  was  asked  a  question  about  the 
applicable  rule,  and  required  to  provide  an  explanation  of  the  decisions  or  actions 
performed.  That  is,  after  the  replay  of  the  mission  segment  containing  the  error, 
in  accordance  with  Bailey  (1982),  the  participant  was  prompted  to:  (a)  state  what 
the  error  was;  (b)  why  it  occurred,  and;  (c)  what  could  be  done  to  correct  it.  Only 
the  most  critical  single  error  from  each  of  the  three  mission  tasks  (BO,  LZ,  and 
OFP)  was  presented  for  discussion  during  the  AAR.  The  AAR  of  critical  errors 
with  coaching  was  the  only  “instruction”  that  the  No  IC/No  AD  Arrow  IF 
(Control)  group  experienced.  No  AAR  was  administered  after  the  fourth  exercise. 

•  Participants  were  required  to  remain  out  of  the  VE  for  a  minimum  twenty  minute 
intervention  period  between  sessions  to  minimize  sickness  and  enable  adaptation. 
As  noted  above,  a  PQ  was  also  administered  after  the  first  and  fourth  VE  session. 
This  intervening  period  was  typically  occupied  by  the  SSQ,  the  AAR,  and 
preparation  for  the  following  exercise. 

9.  After  the  four  mission  cycle  was  completed,  the  purpose  of  the  experiment  was  again 
reviewed  with  the  participant  and  compensation  for  participation  in  the  experiment  was 
provided.  Before  being  released  at  twenty  minutes  after  exiting  the  last  VE  experience, 
the  participant  completed  a  final  SSQ  to  ensure  complete  recovery  from  their  VE 
experiences. 

Participants  were  not  constrained  to  time  limits  for  the  three  individual  tasks  during  the 
exercises;  although  they  were  given  a  one  minute  warning  before  the  twelve  minute  session  limit 
was  reached.  All  participants  were  stopped  at  the  twelve  minute  limit,  sometimes  without 
completing  all  of  the  tasks  (see  Results,  below). 

To  recapitulate,  each  participant  proceeded  through  four  map  and  VE  exercises  during  the 
experiment,  with  differing  VE  terrain  databases  (two  different  MOUT  sites,  each  with  two 
widely  spaced  starting  points)  being  presented  in  counter-balanced  order  while  following  the 
same  process  and  protocol  for  each  reconnaissance  exercise.  The  counter-balancing  structure 
presented  each  of  the  four  environments  in  each  sequential  position  an  equal  number  of  times. 
The  counter-balancing  also  insured  that  participants  never  experienced  the  same  MOUT 
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environment  sequentially.  This  counter-balancing  produced  eight  unique  sequences,  which  were 
then  used  as  the  basis  for  the  number  of  participants  in  each  condition.  For  all  participants,  the 
fourth  mission  was  the  no-intervention  test  mission,  and  as  noted  above  no  AAR  was 
administered  after  the  fourth  (test)  mission. 


Results 

Questionnaire  Data  Analyses 

The  most  basic  use  of  the  biographical  data  is  to  discover  any  possible  intervening  issues 
that  the  participants  bring  to  the  research  that  would  affect  the  analysis  of  the  independent 
variables  in  the  experiment.  General  Linear  Model  analyses  (i.e.  2x2  ANOVA)  of  the 
participant’s  basic  characteristics  and  prior  experiences  found  no  significant  differences  from 
any  of  the  reported  biographical  information  between  the  major  factors  (Interrogative  Coaching 
or  Attention  Direction)  in  this  experiment.  The  important  descriptive  information  about  the 
participant  sample  was  presented  in  the  Participants  section,  above. 

As  introduced  above,  the  ITQ  is  intended  to  provide  information  about  qualities  and 
tendencies  of  participants  in  the  evaluation  of  their  environmental  experience  that  may  influence 
or  explain  performance  in  the  simulated  environment.  The  first  analysis  is  the  same  as 
performed  on  the  biographical  data,  a  General  Linear  Model  analysis  examining  differences  in 
the  ITQ  scales  between  the  major  factors  (Interrogative  Coaching  and  Attention  Direction)  in  the 
experiment.  No  significant  differences  in  the  ITQ  scores  or  scales  were  found  between  the  major 
factors  and  interactions  used  in  this  experiment. 

The  PQ  is  intended  to  directly  address  the  subjective  experience  in  the  simulation 
environment,  and  unlike  the  biographical  or  ITQ,  was  administered  at  several  different  points  in 
the  experimental  sequence  (see  Procedures,  above).  In  order  to  establish  whether  there  was  any 
change  in  the  PQ  that  could  be  directly  attributed  to  the  factors  used  in  the  experiment  a 
repeated-measures  General  Linear  Model  was  performed  in  similar  fashion  to  the  ITQ  and 
biographical  analyses  discussed  above,  with  the  addition  of  the  repeated  administrations  of  the 
PQ.  While  the  PQ  did  change  significantly  over  the  three  repeated  administrations  (F(2, 54)  = 
14.172,  p<.001),  no  significant  differences  were  found  between  the  factors  or  factor  interactions. 
The  PQ  decreased  from  the  first  administration  following  post-movement  training  (M=98.53; 
averaged  over  all  conditions),  to  the  post-mission  one  administration  (M=95.65),  and  increased 
following  the  fourth  or  test  mission  (M=l  06.84).  These  results  do  not  appear  to  provide  any 
additional  explanation  of  the  experiment  factor  results  presented  below,  and  therefore  no  further 
analyses  or  discussion  of  the  PQ  results  are  presented. 

As  explained  in  the  procedures  section,  above,  the  SSQ  was  administered  before  and  after 
each  VE  session,  including  the  movement  and  equipment  operations  training  that  began  the 
experiment  sequence.  The  data  of  interest  for  the  performance  analysis  is  whether  there  was  a 
pattern  in  the  changes  (post-test  minus  pretest)  over  the  experimental  missions.  A  repeated- 
measures  General  Linear  Model  analysis  was  performed,  which  found  no  significant  differences 
over  the  four  mission  differences,  between  the  factors  (Interrogative  Coaching  and  Attention 
Direction)  or  the  factor  interactions.  This  is  not  totally  unexpected,  as  anyone  experiencing  an 
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abnormally  high  SSQ  score  before  any  VE  session  was  not  allowed  to  participate  (rated  as  being 
in  the  upper  ten  percent  based  on  Kennedy,  et  al.,  1 993).  In  addition,  any  participant 
experiencing  a  large  increase  in  their  SSQ  score  post-mission  (the  rule  of  thumb  used  was  a 
doubling  in  their  score)  either  had  to  recover  rapidly  and  not  experience  a  repeated  similar 
episode  or  withdraw  from  the  experiment  (receiving  credit  for  participation  to  that  point).  A 
total  of  five  participants  were  withdrawn  from  the  experiment  based  on  these  conditions. 

Performance  Measurement 

As  these  are  complicated  activities,  there  were  several  process  and  outcome  measures  that 
were  used  for  analysis  of  the  VE  instructional  interventions.  Each  of  the  three  tasks  (Bounding 
Overwatch  [BO],  Landing  Zone  placement  [LZ],  and  Observation/Fire  Post  placement  [OFP]) 
was  measured  separately,  and  in  several  ways.  The  task  measures  included;  task  performance 
times,  time  spent  inspecting  buildings  during  the  task,  number  of  buildings  inspected,  and  rated 
performance  scores.  The  task  performance  times  and  rated  performance  scores  are  outcome 
measures,  while  the  building  inspection  times  and  numbers  should  reflect  rule  application 
processes  associated  with  the  tasks.  The  rationale  for  using  these  measures  is  that  they  should 
change  based  on  the  efficacy  of  the  instructional  approach  and  IF  being  used.  The  following 
situation  is  provided  as  an  example  of  the  instructional  approach  in  applying  both  of  the 
instructional  features.  The  Experimenter  would  use  the  Directional  Arrow  to  point  at  a  building 
that  should  be  inspected  (based  on  rules  provided  during  instruction,  see  Appendix  B),  while 
asking  a  question  (Interrogative  Coaching  application)  about  the  appropriate  rule.  This  should 
increase  the  application  of  the  rule  by  the  participant,  and  improve  the  correct  performance  of 
that  aspect  of  the  task. 

Performance  Time.  The  VE  system  was  set  up  to  make  a  record  of  all  important  events 
that  occurred  during  each  second.  Total  mission  time  recording  began  (controlled  by  the 
Experimenter)  when  the  participant  was  told  to  begin  the  mission  and  ended  when  the  participant 
indicated  that  they  were  finished,  or  the  Experimenter  enforced  the  twelve  minute  time  limit. 
Because  there  was  unavoidable  lag  time  at  both  the  beginning  and  end  of  every  session,  we 
measured  performance  time  data  from  the  time  that  the  first  mark  was  made  (usually  a  BO  mark) 
until  the  time  that  the  last  mark  made  (almost  always  an  OFP  mark).  Data  reduction  routines 
were  used  to  track  and  compile  time  spent  on  each  task  based  upon  the  marks  used  and  the 
recorded  times.  Post-processing  software  accumulated  the  time  between  making  the  marks,  and 
summed  those  intervals  based  upon  the  preceding  type  of  mark,  so  that  total  performance  times 
for  each  separate  task  was  calculated  even  if  the  task  performances  overlapped. 

Building  Inspection  Time.  As  environment  inspection  was  an  important  part  of  every  task 
(based  on  the  task  instructions,  see  Appendix  B),  and  the  environments  were  urban,  building 
inspection  by  the  participant  during  each  task  was  also  used  as  a  measure  for  the  task.  Buildings 
were  considered  to  be  under  inspection  during  any  time  interval  if  fifty  percent  of  any  face  of  the 
building  was  in  the  central  visual  area  of  the  HMD  (calculated  by  vector  analyzing  (invisible) 
points  on  each  face  of  a  building  during  post-processing  of  the  participant  visual  field  from  each 
mission).  In  this  manner,  at  distance,  several  buildings  could  be  counted  as  being  inspected 
simultaneously  during  a  one  second  interval.  In  order  to  better  measure  inspection,  rather  than 
the  number  of  times  that  a  building  was  viewed  for  even  a  part  of  a  second,  the  decision  was 
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made  to  only  include  the  building  if  it  was  in  the  FOV  for  three  consecutive  seconds.  The 
amount  of  time  spent  inspecting  buildings  was  the  accumulation  of  seconds  with  one  or  more 
buildings  in  view  (for  at  least  three  consecutive  seconds)  during  performance  of  a  particular  task. 

Number  of  Buildings  Inspected.  As  with  building  inspection  time,  the  total  number  of 
buildings  inspected  was  determined  by  calculating  intervals  of  consecutive  3-second  gazes  at  an 
individual  building  and  counting  that  as  a  single  building  inspection.  In  addition,  a  building 
would  have  to  be  within  the  central  visual  area  of  the  HMD  during  another  three  second  interval 
to  be  counted  as  another  building  being  inspected.  This  is  based  on  the  idea  that  one  should 
count  repeated  inspections  of  the  same  building,  as  they  may  have  been  inspected  from  different 
positions  or  for  different  reasons.  Again,  multiple  buildings  could  be  under  inspection  at  any 
time,  and  as  noted  above  buildings  could  be  repeatedly  inspected.  The  number  of  building 
inspections  meeting  these  criteria  during  a  particular  task  was  then  the  sum  of  times  that  each 
building  was  inspected  based  upon  time  and  FOV. 

Task  Performance.  Task  performance  was  derived  by  rating  the  correct  application  of 
applicable  rules  for  the  different  aspects  of  each  task  (see  Appendix  B  for  the  trained  rules).  BO 
positions  were  scored  on  spacing  distance  between  positions,  cover,  and  concealment  available  at 
those  positions,  and  visual  overlapping  fields  of  fire  between  the  current  and  previously  marked 
position.  The  BO  score  was  then  summed  (over  each  category)  and'averaged  (over  categories)  to 
generate  a  rating  for  each  participant.  LZ  positions  (for  a  helicopter)  were  scored  on  clearance 
from  obstacles,  proximity  to  village  center,  and  the  correctness  of  size  and  shape.  These  ratings 
were  then  summed  and  averaged  to  generate  a  rating  for  each  participant.  OFP  positions  were 
scored  based  on  cover  and  concealment,  coverage  of  the  LZ,  and  visual  overlapping  fields  of  fire 
between  the  positions.  The  scores  were  summed  and  averaged  over  positions  to  generate  a  rating 
for  each  participant  on  the  OFP  task.  Final  performance  scores  were  determined  by  averaging 
three  Experimenter/Subject  Matter  Experts  (SMEs)  ratings  on  how  well  the  rules  for  task 
performance  on  these  described  characteristics  were  followed  for  egch  task.  Inter-rater  reliability 
(a)  for  the  ratings  on  the  test  (fourth)  mission  ranged  from  a  high  of  .96  to  a  low  of  .63  for  these 
ratings,  which  was  deemed  acceptable. 

Missing  Data.  Because  the  performance  of  the  tasks  within  the  missions  was  not  held  to  a 
performance  timeline  (although  the  mission  itself  was  limited  to  twelve  minutes),  there  were  a 
few  instances  in  which  LZ  tasks  were  not  completed,  and  a  larger  number  in  which  the  OFP  task 
(typically  performed  last  in  the  sequence)  was  not  performed.  Two  participants  in  the  first 
mission  and  two  in  the  third  mission  did  not  sufficiently  complete’ the  LZ  phase  forjudging. 
These  missing  data  were  replaced  using  the  mean  of  the  condition  for  each  of  those  participants 
for  those  missions.  The  condition  mean  was  used  as  replacement  data  in  order  to  minimize  the 
possible  distortion  of  the  results  for  the  experimental  conditions,  while  maintaining  requirements 
for  the  analysis  of  variance.  Condition  or  cell  means  were  not  considered  to  bias  the  analysis  of 
the  individual  factors,  maintained  adequate  representation  in  each  group,  and  were  considered  to 
be  more  representative  than  factor  means  or  the  mean  for  the  mission  (over  all  factors). 

For  the  OFP  task,  five  participants  did  not  perform  the  task  at  all  on  the  first  mission,  seven 
did  not  perform  that  task  on  the  second  mission,  three  did  not  perform  the  task  on  the  third 
mission,  and  two  did  not  perform  the  task  in  the  fourth  mission.  One  participant  did  not  perform 
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the  task  during  any  of  the  four  missions.  Because  this  was  a  large  amount  of  missing  data,  and 
replacement  using  overall  experimental  means  would  have  precluded  reasonable  analyses,  no 
analyses  were  performed  on  the  OFP  task  data. 

Data  Normality.  Explorations  of  the  BO  and  LZ  rating  data  were  conducted  to  check  for 
normality  and  possible  confounding.  The  BO  and  LZ  performance  ratings  were  found  to  be  non¬ 
normal  over  almost  all  missions  in  that  all  of  the  data  were  negatively  skewed  and  a  majority  of 
the  mission  data  also  had  large  kurtosis  (using  the  Kolmogorov-Smimov  test  in  SPSS  vs.  12,  as 
well  as  the  standard  z  transformation  of  the  skewness  and  kurtosis  values  for  the  data).  The 
recommendation  by  Tabachnick  and  Fidell  (1996)  for  data  sets  with  these  characteristics  is  to 
either  adjust  outliers  that  are  driving  the  non-normality  (deemed  an  unacceptable  approach  with 
these  data),  transform  the  entire  data  set  to  reduce  or  eliminate  the  non-normal  characteristics,  or 
both.  The  recommended  sequence  for  adjustment  for  transformation  of  such  data  sets  is  to  first 
inspect  the  data  for  potential  intervening  factors  that  could  be  contributing  to  the  outlying  data 
points,  then  transform  the  data  based  upon  the  type  and  amount  of  non-normal  characteristics 
(skewness  and  kurtosis),  then  inspect  for  and  correct  as  necessary  any  remaining  outliers 
(Tabachnick  &  Fidell,  1996).  The  process  followed  with  this  data  set  is  described  in  the 
following  paragraphs. 

As  noted  earlier  in  connection  with  the  questionnaire  data  analysis,  there  were  no 
significant  relationships  found  between  that  data  and  the  experiment  performance  data  that 
would  account  for  the  skewed  and  kurtosed  distributions.  Given  the  relatively  consistent  shape 
of  the  distributions  of  the  performance  data  (negatively  skewed  and  positively  kurtosed),  and  the 
fact  that  outliers  did  not  seem  to  be  “driving”  the  distribution  of  the  conditions,  the  performance 
rating  data  for  BO  and  LZ  missions  were  transformed  using  the  sequence  recommended  by 
Tabachnick  and  Fidell  (1996).  The  data  were  reflected  (each  value  was  subtracted  from  the 
maximum  possible  value  plus  1 .0)  in  order  to  make  the  data  positively  skewed  (with  a  minimum 
value  of  1 .0).  The  reflected  data  were  then  logarithmically  transformed,  to  reduce  both  the 
skewness  and  kurtosis.  This  reduced  the  non-normal  characteristics  for  the  BO  and  LZ 
performance  data,  leaving  only  the  data  for  mission  4  with  any  significant  kurtosis  in  any 
condition.  The  transformation  did  leave  some  outliers  remaining  in  the  LZ  data,  but  it  seemed 
more  reasonable  to  leave  these  cases  untouched  as  there  were  no  data-driven  rationales  for 
adjusting  them,  and  the  distributions  were  no  longer  significantly  different  from  normal  at  the 
overall  mission  level  (with  the  exception  mentioned  above). 

Following  the  reflection  and  transformation  of  the  data,  another  outlier  inspection  was 
performed.  The  questionnaire  data  was  used  to  check  for  possible  intervening  factors  that  might 
be  contributing  to  any  remaining  outliers.  An  inspection  of  likely  candidate  factors  found  no 
correlations  between  presence  or  immersive  tendencies  and  the  performance  ratings  for  the  two 
tasks.  Many  of  the  other  factors  of  interest  were  not  suitable  for  examination  through 
correlation,  as  high  percentages  of  the  data  were  anchored  with  zero  occurrences  or  ratings.  A 
visual  examination  of  that  data  found  that  for  two  relatively  consistent  outlier  cases  in  the  LZ 
ratings,  consistent  but  high  simulator  sickness  ratings  had  been  collected,  although  the  levels 
never  reached  the  criteria  used  for  dismissal  from  the  experiment.  These  outliers  were  adjusted 
on  the  basis  of  possible  negative  effects  from  simulator  sickness  on  their  performance  in  the 
experiment.  The  adjustment  technique  adopted  was  as  follows:  using  the  original  data,  the 
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highest  outlier  value  for  these  cases  was  changed  to  one  interval  less  than  the  next  score  (closer 
to  the  mean  for  that  cell).  For  example,  a  case  with  an  outlier  value  of  8  was  changed  to  12.5, 
with  the  next  higher  value  in  the  distribution  being  13.5.  Following  this  adjustment,  the  data 
were  again  reflected  and  logarithmically  transformed  for  analysis.  This  technique  is  suggested  in 
order  to  maintain  the  greatest  amount  of  information  in  the  data  while  decreasing  the 
confounding  influence  of  that  case  on  the  distribution  of  the  data  (Tabachnick  &  Fidell,  1996). 

Experiment  Data  Analyses 

In  order  to  test  for  differences  between  the  experimental  factors  over  multiple  missions,  a 
repeated  measures  General  Linear  Model  (i.e.  MANOVA,  using  SPSS  vs.  12)  analyses  was 
conducted  for  each  task  over  the  four  repeated  missions  with  IC  (2  levels)  and  AD  Arrow  (2 
levels)  factors.  Separate  analyses  were  performed  for  each  task  using  the  performance  ratings 
(transformed  as  described  above),  performance  times,  time  spent  inspecting  buildings,  and 
number  of  buildings  inspected.  Analyses  were  performed  separately  on  the  tasks  and  measures 
because  there  was  no  reason  to  combine  the  BO  and  LZ  tasks  for  analysis  purposes,  nor  was 
there  a  relationship  between  performance  ratings,  performance  time,  time  spent  inspecting 
buildings,  or  number  of  buildings  inspected.  In  fact,  combining  some  of  these  dependent 
variables  would  have  violated  independence,  as  the  time  measures  are  related.  The  significant 
results  of  these  several  analyses  are  therefore  presented  individually,  and  non-significant  results 
are  not  reported  in  detail  at  all. 

Bounding  Overwatch.  The  BO  Performance  transformed  data  still  evidenced  homogeneity 
problems  between  conditions  (indicated  by  a  significant  Mauchly’s  W  (W(5)=.520;  p=.004),  and 
therefore  the  Huynh-Feldt  adjusted  degrees  of  freedom  (provided  by  SPSS  vs.  13)  were  used  to 
compensate  for  the  non-homogeneity.  This  analysis  for  BO  performance  found  significant 
differences  over  Missions  (F(2.444168.43i)=l 3.273;  p<.001;  Power  =  .999),  Missions  X  Coaching 
(F(2.444,68.43i)=3.806  p=.020,  Power  =  .735)  and  Missions  X  AD  Arrow  (F(2.444,68.43i)=2.923; 
p=.050,  Power  =  .610).  The  means  for  these  data  over  missions  are  presented  in  Table  3. 

The  univariate  analysis  of  the  between  subjects  conditions  for  Coaching  and  AD  Arrow 
were  not  significant  using  the  transformed  data  averaged  over  the  four  missions.  The  means  over 
missions  and  conditions  for  both  the  original  and  the  transformed  BO  performance  ratings  are 
presented  in  Table  3.  (Note  that  for  the  transformed  data,  lower  values  reflect  better 
performance.  It  should  also  be  noted  that  outlier  corrections  were  not  used  in  calculating  the 
untransformed  means.  Finally,  although  all  four  missions  were  used  in  the  analyses,  and  are 
presented,  the  reader  should  remember  that  no  treatment  was  applied  to  any  group  during  the 
fourth  mission.) 

As  noted  above,  a  separate  analysis  was  performed  using  the  BO  Performance  Time  data 
(not  transformed,  see  the  bottom  entries  in  Table  3),  with  a  significant  result  for  Missions  (Fp, 
84)=5.017  p=.003,  Power  =  .904).  The  instructional  factors  did  not  significantly  interact  with 
overall  Mission  times,  nor  were  they  significant  for  BO  Performance  Time  when  averaged  over 
all  four  missions.  Separate  but  similar  analyses  of  the  Number  of  Buildings  Inspection  and  Time 
Spent  Inspecting  Buildings  measures  revealed  no  significant  effects,  and  these  analyses  and  data 
are  therefore  not  presented. 
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Table  3. 

Actual  and  Transformed  Mean  Bounding  Overwatch  Performance  Ratings  &  Times  over 
Missions  &  Conditions 


Missions 

One 

Two 

Three 

Four 

BO  Performance  Ratings  * 

30.6507 

32.3640 

33.8477 

34.2201 

Transformed  Data 

.6712 

.5864 

.4421 

.3764 

Coaching  Condition* 

30.2651 

33.0874 

33.7949 

34.9659 

Transformed  Data 

.7208 

.5014 

.4307 

.2684 

No  Coaching  Condition* 

31.0363 

31.6407 

33.9005 

33.4744 

Transformed  Data 

.6217 

.6714 

.4535 

.4844 

Attention-Direction  Condition* 

30.1361 

32.7985 

33.0342 

34.0231 

Transformed  Data 

.6776 

.5355 

.5435 

.4033 

No  Attention-Direction  Condition* 

31.1653 

31.9296 

34.6613 

34.4172 

Transformed  Data 

.6649 

.6373 

.3407 

.3495 

BO  Performance  Times  (seconds) 

294.6250 

272.9688 

259.0000 

224.8125 

*Non-transformed  data  without  outlier  adjustment  are  presented  for  clarity  of  presentation. 


Landing  Zone.  A  repeated  measures  analysis  of  the  transformed  LZ  performance  data 
found  significant  differences  over  the  repeated  Missions  (F(3ig4)=5.479,  p=.002,  Power=.929). 

No  other  tests  were  significant  for  the  within  subjects  analysis.  The  actual  and  transformed 
means  for  the  significant  factors  are  presented  in  Table  4. 

The  between  subjects  test,  with  transformed  data  averaged  over  missions,  found  the  AD 
Arrow  significant  (F(i>28)=5.794,  p=.023,  Power=.642).  In  this  situation,  AD  Arrow  mean  was 
.979  (SE=.033)  and  No-Arrow  Mean  was  .868  (SE=.033),  indicating  that  not  using  the  AD 
Arrow  provided  generally  better  performance  on  the  LZ  task.  LZ  performance  times  were  also 
significantly  different  over  Missions  (F(3;  g4)=8.774,  P<.001 ,  Powep=.993),  and  significantly 
different  over  Mission  X  Coaching  (Fn  g4y=2.826.  p=.044,  Power=.660;  see  Table  4). 

In  the  (averaged  over  missions)  univariate  analysis  of  performance  times  between  subjects 
effects  Interrogative  Coaching  was  again  significant  (F(i  26)=4.654,  p=.04,  Power=.549;  see  Table 
5,  overall  times).  Separate  analyses  of  the  observation  and  time-based  measures  did  not  find 
significant  effects,  and  these  data  are  therefore  not  presented. 

Test  Mission  Analyses.  Separate  ANOVAs  were  also  conducted  on  the  Performance  Time, 
Building  Inspection  Time,  Number  of  Buildings  Inspected,  and  transformed  Performance  scores 
for  both  the  BO  and  LZ  tasks  for  only  the  fourth  (test)  mission.  The  preliminary  tests  for  the 
univariate  analysis  found  that  the  group  variances  were  unequal  for  the  transformed  BO 
Performance  data  (Levene’s  F(3j2g)=4.39;  p<.012)  but  not  for  any  other  data.  Only  the 
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transformed  BO  Performance  for  Interrogative  Coaching  was  found  to  be  significant  in  the 
General  Linear  Model  univariate  analysis  (F(ij28)=7.099,  p=.013,  power=.730).  A  non-parametric 
analysis  of  the  transformed  performance  score  for  the  final  mission  BO  phase  also  found  that  it 
differed  significantly  over  the  Coaching  condition  (Mann- Whitney  U=66.0,  p=. 019; 

Interrogative  Coaching  Mrank- 12.63,  No-Coaching  Mrank-20.38;  note  that  as  the  data  were 
reflected  and  log  transformed  the  lower  rank  represents  better  performance). 


Table  4. 

Actual  and  Transformed  Mean  Landing  Zone  Performance  Ratings  &  Times  over  Missions  & 
Conditions 


Missions 

One 

Two 

Three 

Four 

LZ  Performance  Ratings 

25.3307 

26.4888 

28.50 

28.6016 

Transformed  Data* 

1.0005 

.9580 

.8863 

.8483 

LZ  Times  (overall) 

268.8393 

216.0089 

223.6250 

179.0938 

Coach  Condition 

310.6786 

229.3304 

266.9375 

180.5000 

Overall  Mission  Mean=246.862 
No  Coach  Condition 

227.0000 

202.6875 

180.3125 

177.6875 

Overall  Mission  Mean=l  96.922 
*Note:  Outliers  were  adjusted  before  data  transformation. 


Discussion 

The  primary  goal  of  this  research  was  to  assess  the  benefits  of  two  VE-based  instructional 
features:  oral  interrogative  coaching  and  a  simple,  direction-indicating  arrow  (which  pointed  at 
cues  critically  related  to  performed  errors).  Coaching,  whether  in  interrogative  or  directive  form, 
serves  to  provide  feedback  about  performance  as  well  as  information  and/or  cues  about  improved 
or  correct  performance.  In  a  similar  fashion,  if  an  AD  Arrow  points  at  some  stimulus  in  the 
environment,  the  presumption  is  that  some  aspect  of  performance  is  less  than  optimal  and  the 
indicated  stimulus  is  related  to  better  performance  (as  noted  in  the  protocol  instructions). 
Obviously,  when  Coaching  is  combined  with  some  AD  device,  the  presumption  is  that  more 
information  has  been  provided  about  how  to  improve  performance.  In  this  way,  the  coaching 
intervention  frames  the  Attention-Direction  indicated  stimulus  and  an  interaction  effect 
(improvement  over  the  presentation  of  either  intervention  alone)  might  be  expected. 

The  significant  improvements  in  both  performance  and  the  time  to  perform  found  for  both 
the  BO  and  LZ  phases  over  the  repeated  Missions  provide  adequate  evidence  of  learning.  The 
basic  patterns  for  the  time  to  perform  the  tasks  is  relatively  clear,  with  steadily  decreasing 
amounts  of  time  required  for  steadily  improving  (overall)  performance.  The  conclusion  that 
participants  learned  over  the  repeated  missions  is  not  startling,  but  does  provide  a  solid  basis  for 
interpreting  the  results  of  interest. 
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Increased  support  for  the  efficacy  of  an  Instructional  Feature  would  be  provided  if  that 
treatment  demonstrated  a  more  rapid  learning  than  the  “control”  condition  of  no  instructional 
intervention,  as  would  be  indicated  by  a  Mission  by  IF  interaction.  The  significant  interaction 
with  BO  data  (transformed)  for  Missions  with  Coaching  seems  to  indicate  that  questions  leading 
to  or  reminding  participants  about  the  rules  that  guide  performance,  administered  during  the 
training  exercises  supports  improved  performance  (see  Table  3),  which  is  inferred  as 
representing  improved  learning  for  those  receiving  that  intervention.  The  data  pattern  indicates 
that  Coached  performance  started  off  slightly  worse  than  non-Coached,  but  steadily  improved 
(decreasing  transformed  values).  This  pattern  seems  to  provide  an  understandable  explanation 
for  the  results  of  this  analysis  (note  that  no  post  hoc  analysis  of  the  repeated  measures  and 
between-subjects  factor  can  be  performed). 

There  was  also  a  significant  interaction  in  the  BO  data  for  Missions  with  the  AD  Arrow, 
and  these  transformed  data  also  have  an  understandable  pattern  of  results  (see  Table  3).  There  is 
general  improvement  over  the  missions,  but  in  this  situation  NOT  providing  an  AD  Arrow 
intervention  provides  better  improvement.  So  for  some  reason,  pointing  out  stimuli  relevant  to 
the  application  of  the  task  performance  rules  did  not  improve  performance.  Given  the 
experimental  structure,  half  of  the  group  shown  an  AD  Arrow  did  receive  coaching,  as  did  half 
of  those  not  shown  an  AD  Arrow.  The  conclusion  must  be  that  the  AD  intervention  did  not 
provide  any  extra  clarifying  information  that  could  support  skill  acquisition,  and  actually  delayed 
performance  improvements  over  the  missions. 

Significant  interactions  were  not  found  in  the  transformed  LZ  performance  data,  but  were 
found  with  the  time  to  perform  the  LZ  task.  The  transformed  LZ  performance  data  did  find 
performance  improvements  over  Missions  (see  Table  4),  indicating  that  participants  did  learn  to 
perform  the  task  better  over  trials.  The  analysis  of  Performance  Time  data  found  significant 
improvement  over  Missions,  and  a  significant  interaction  with  the  Interrogative  Coaching 
intervention  (see  Table  4).  The  non-Coached  group  performed  more  quickly  overall  than  the 
Coached  participants,  although  the  Interrogatively  Coached  participants  performed  almost  as 
rapidly  in  the  fourth  or  Test  Mission. 

The  Mission  4  (the  test  mission)  analysis  investigated  any  outcomes  from  the  different 
conditions  found  during  a  non-interventional  (no  IF  applied)  final  trial.  This  trial  was  expected 
to  show  the  final  level  of  skill  acquired  from  the  preceding  mission  interventions  and  AAR 
sessions.  The  only  finding  in  this  data  was  a  significant  effect  on  BO  performance  from  the 
Coaching  intervention  (both  with  the  ANOVA  and  the  confirmatory  Non-parametric  test  on  that 
specific  condition). 

Overall,  these  results  seem  to  be  consistent  with  Bailey’s  (1982,  p.  326)  conclusion  that 
“speech  may  be  preferred  when  the  message  calls  for  immediate  action,  vision  is  already 
overburdened,  or  the  job  requires  the  user  to  move  about  continually.”  The  speech  in  this  case, 
while  interrupting  the  task  activity,  was  also  providing  some  learning  benefit  for  subsequent 
performances.  The  visual  system  was  already  overloaded  with  task  activities,  hence  the  lack  of 
significant  improvement  from  the  addition  of  an  attention  direction  mechanism.  However,  this 
conclusion  is  weakened  since  these  factors  were  characteristics  of  the  LZ  task  as  well.  Not 
finding  this  difference  in  the  LZ  task,  prevents  the  generation  of  clearly  supported  guidance. 
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Further  analysis  of  the  tasks  might  provide  insight  into  the  learning  pattern  found  in  this 
research.  However,  that  analysis  is  probably  not  an  efficient  approach  because  training  tasks 
through  the  use  of  coaching  and  attention  direction  mechanisms  like  these  is  seldom  restricted  to 
the  experimental  controls  necessarily  used  in  this  research.  Normal  BO  and  LZ  training  would 
encompass  more  classroom  instruction  and  a  wider  variety  of  field  practice,  which  would  not  be 
limited  to  a  single  coaching  or  attention-direction  intervention  during  each  exercise. 

An  inspection  of  the  data  (see  Tables  3  &  4,  above)  reveals  that  more  time  was  expended 
during  the  training  missions  on  the  BO  task,  relative  to  the  LZ  task  (see  above  comments  about 
non-performance  of  the  OFP  task,  which  also  provides  support  for  the  following  argument).  The 
difference  in  time  to  perform  could  support  the  argument  that  improvement  in  the  BO  task  was 
not  possible  in  the  time  available  for  the  LZ  task.  This  would  also  mean  that  results  similar  to 
those  found  with  the  BO  task  might  be  expected  for  the  LZ  task  if  comparable  time  for  rule 
applications  and  cognitive  effort  were  expended  in  the  training  exercises. 

The  possibility  also  exists  that  the  application  of  the  AD  Arrow  IF  was  not  well  understood 
by  some  participants,  regardless  of  the  repeated  instructions  (participants  were  reminded  of  the 
experimental  interventions  and  intent  prior  to  each  trial  for  that  specific  condition).  In  the  AD 
Arrow  only  condition,  the  directional  stimulus  would  point  toward  the  environmental 
configuration  that  should  have  changed  the  decision-based  performance  (as  the  stimulus  was 
only  implemented  following  an  error).  It  may  have  been  that  although  the  participants  looked  in 
the  indicated  direction  as  instructed,  that  the  additional  visual  inspection  during  their  tasks  may 
not  have  supported  the  desired  change  in  understanding  (an  implicit  feedback  situation).  In  this 
situation,  better  performance  could  have  been  based  on  the  un-interrupted  application  of  those 
rules  that  were  understood.  So  the  AD  Arrow,  alone  or  combined  with  Interrogative  Coaching, 
could  have  interfered  just  enough  with  LZ  performance  that  the  non-AD  Arrow  condition  (which 
combined  both  non-coached  and  coached  without  presentation  of  an  AD  arrow)  evidenced 
enough  improved  performance  to  preclude  finding  any  significant  difference.  (This  is  not  an 
argument  that  the  conditions  were  equivalent  based  upon  finding  no  significant  differences,  but  a 
possible  set  of  conditions  that  lead  to  not  finding  significant  differences.) 

Finally,  and  in  retrospect,  it  seemed  that  the  time  required  for  the  question-based  coaching 
or  AD  Arrow  interventions  during  the  missions  could  have  actively  taken  time  and  cognitive 
resources  away  from  those  trainees,  relative  to  others  in  the  other  conditions.  This  would  have 
left  those  with  the  highest  level  of  intervention  (both  IFs)  with  less  time  and  cognitive  resources 
for  actually  learning  to  perform  the  task,  although  they  presumably  could  have  had  improved 
mental  models  of  the  required  task  performance  parameters.  Some  minimal  support  for  this 
interpretation  may  be  derived  from  the  BO  phase  analysis  results,  in  that  overall  the  individual 
factors  supported  significant  improvement,  while  the  combined  factors  did  not. 

These  initial  results  from  this  line  of  research  indicate  that  injecting  more  information  into 
the  virtual  training  environment  does  not  necessarily  translate  into  better  learning  performance 
for  all  tasks.  The  results  do  indicate  an  interrogative-style  coaching  instructional  feature  during 
the  VE  mission  can  be  beneficial  in  the  initial  acquisition  of  an  important  environment- 
evaluation  based  decision  activity.  What  is  not  clear  is  whether  that  form  of  coaching 
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intervention  is  best  for  initial  skill  acquisition  on  different  types  of  tasks.  Obviously,  care  must 
be  taken  in  using  any  interventional  instructional  feature  or  performance  decrements  may  result. 
The  results  of  this  and  future  research  (see  below)  will  be  used  to  provide  guidelines  for  the 
implementation  and  use  of  instructional  features  in  simulation-based  training. 

Further  Research 

A  second  instructional  feature  experiment  has  been  conducted  to  explore  and  extend  the 
findings  of  this  first  experiment.  The  design  and  variables  were  changed  as  little  as  possible 
from  the  original  experiment  in  order  to  leverage  insights  and  resources  across  experiments. 
However,  the  following  presentation  and  methodological  changes  have  been  made. 

1 .  An  additional  independent  variable  has  been  added  to  the  experimental  design  -  a  text 
window  with  a  coaching  feedback  message  that  matches  the  auditory  coaching  intervention. 

Like  the  other  instructional  features,  the  text  window  appears  (under  control  of  the  experimenter) 
in  the  FOV  immediately  after  the  first  error  of  each  mission  phase.  Similar  approaches  have 
proven  successful  in  simulator-based  flight  training  (Skitka,  Mosier,  &  Burdick,  1999;  Loftin, 
Wang,  Baffes,  &  Hua,  1992). 

2.  The  form  of  the  auditory  intervention  to  participants  has  been  changed  from  an 
interrogative  question-based  to  a  specific  directive-based  intervention.  The  coaching  has  the 
following  format:  (a)  you  have  done  something  wrong;  (b)  here  are  the  parameters  on  how  to  do 
it  right,  and;  (c)  do  the  task  activity  again. 

3.  Strict  time  limits  were  enforced  for  each  of  the  three  mission  phases  or  tasks.  This  was 
done  to  ensure  that  time  pressure  was  applied  equivalently  for  each  of  the  three  tasks,  and  that 
each  of  the  three  tasks  was  addressed  by  every  participant. 

4.  The  experimenter-standardized  auditory  directive  coaching  was  recorded  and  has  been 
used  in  a  more  automated  play-back  manner  (again,  under  control  of  the  experimenter)  in  order 
to  mitigate  any  experimenter  time  or  error-based  confounding  during  the  instructional 
interventions.  The  same  directives  are  presented  in  the  text  window  coaching  condition  (see 
above)  and  during  the  AAR,  in  response  to  similar  critical  errors. 

5.  The  length  and  format  of  the  AD  Arrow  IF  stimulus  was  changed  from  a  short  arrow  in 
front  of  the  participant  to  three  converging  lines  that  extend  all  the  way  to  the  error  location  or 
important  stimuli,  as  marked  by  the  experimenter. 

6.  On  the  experimenter’s  scenario  control  monitor,  unseen  by  the  participant,  during  the 
VE  mission,  5-  and  50-meter  circles  surround  participant-made  marks  for  BO,  and  OFP 
positions.  LZ  positions  will  be  surrounded  by  7-  and  50-meter  circles  indicating  the  unique 
distances  for  the  proper  LZ  shape.  These  marks  were  shown  to  the  participant  during  the  post¬ 
mission  AAR’s.  Using  these  circles  was  intended  to  provide  several  advantages.  First,  it 
enabled  the  experimenter  to  make  better  judgments  of  distance-related  errors  and  therefore 
support  better  intervention  timing  during  the  mission  exercise.  Second,  it  provided  more  precise 
visual  feedback  to  participants  during  the  post-mission  exercise  AAR’s,  which  was  intended  to 
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enable  better  understanding  of  both  the  feedback  and  the  distances  involved.  Third,  it  enabled 
more  precise  mission  scoring  for  distance-related  errors  following  the  data  collection  efforts. 
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Appendix  A 


Biographical  Questionnaire 


Research  Participant  Information  Questionnaire  ID _ 

Please  fill  in  the  blank  or  circle  the  appropriate  response. 

1.  What  is  your  age?  _ years 

2.  What  is  your  gender?  female  male 

3.  Are  you  currently  in  your  usual  state  of  good  fitness?  yes  no 

4.  How  many  hours  sleep  did  you  get  last  night?  _ hours 

4a.  Was  it  sufficient?  yes  no 

5.  Indicate  all  medications/substances  you  have  used  in  the  past  24  hours: 

CIRCLE  ALL  THA  T  APPLY 

0  -  none 

1  -  sedatives  or  tranquilzers 

2  -  aspirin,  tylenol,  other  analgesics 

3  -  anti-histamines 

4  -  decongestants 

5  -  other  prescription  drugs 

6  -  other  medications  (please  list: _ ) 

6.  Have  you  ever  experienced  motion  or  car  sickness?  yes  no 

7.  How  susceptible  to  motion  or  car  sickness  do  you  feel  you  are? 

0  1  2  3  4  5  6  7 

not  very  average  very 

susceptible  mildly  highly 

8.  Do  you  have  a  good  sense  of  direction?  yes  no 

9.  How  many  hours  per  week  do  you  use  computers?  _ hours  per  week 

10.  My  level  of  confidence  in  using  computers  is 

1  2  3  4  5 

low  average  high 

11.  I  enjoy  playing  video  games  (home  or  arcade). 

1  2  3  4  5 

disagree  unsure  agree 


A-l 


12.  Iam _ at  playing  video  games. 

1  2  3  4  5 

bad  average  good 

13.  How  many  hours  per  week  do  you  play  video  games?  _ 

14.  How  many  times  in  the  last  year  have  you  experienced  a  virtual  reality  game  or 

entertainment? 

0  1  2  3  4  5  6  7  8  9  10  11  12+ 

1 5.  Do  you  have  a  history  of  epilepsy  or  seizures?  yes  no 

1 6.  Do  you  have  normal  or  corrected  to  normal  20/20  vision?  yes  no 

17.  Are  you  color  blind?  yes  no 
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Appendix  B 


Task  Training  Manual 


INTRODUCTION 


As  described  in  the  experiment  briefing  and  demonstration,  you  will  complete  a 
series  of  planning  exercises  modeled  after  US  Army  planning  operations  in  urban  (city¬ 
like)  environments.  During  each  exercise,  you  will  mark  a  plan  on  a  map  and  then  move 
through  a  virtual  enviomment  (VE)  representing  the  different  towns.  The  exercise 
requires  marking  the  location  for  a  helicopter  landing  zone  (LZ),  planning  movement  of  a 
platoon  to  that  LZ,  and  identifying  locations  for  protecting  the  landing  zone,  called 
observation/fire  posts  (OFP).  In  essence,  you  are  planning  as  if  you  would  lead  a  platoon 
to  execute  the  mission  of  establishing  an  LZ. 

This  manual  presents  the  basics  of  military  operations  in  urban  terrain  (MOUT) 
necessary  to  correctly  plan  for  LZ  placement,  use  a  map  in  planning,  and  the  three  types 
of  marks  you  will  use  in  the  exercises.  In  the  first  section,  movement  through  MOUT  and 
LZ  requirements  are  explained,  including  how  to  mark  locations  in  this  research.  In  the 
second  section,  you  will  leam  the  basics  of  how  to  read  military  maps,  including  contour 
lines  and  color  codes.  After  training  you  will  be  given  a  multiple  choice  test  on  this 
information.  Then  you  will  practice  planning  movements  and  marking  locations  on  a 
terrain  map.  You  must  leam  and  demonstrate  the  basic  information  in  this  manual  in 
order  to  participate  in  the  experiment. 

Military  Operations  in  Urban  Terrain 

Built-up  or  urban  areas  consist  mainly  of  man-made  features  such  as  buildings  and 
roads.  Buildings  provide  cover  and  concealment,  limit  fields  of  observation  and  fire,  and 
block  the  movement  of  troops.  Streets  are  usually  avenues  of  approach.  However,  forces 
moving  along  streets  are  threatened  by  the  buildings  and  have  little  space  for  cover. 

Land  navigation  is  important.  There  are  four  steps  to  land  navigation.  Having  an 
objective  and  the  requirement  to  move  there,  you  must  know  where  you  are,  how  to  plan 
the  route,  how  to  stay  on  the  route,  and  how  to  recognize  the  objective. 

Step  1:  Know  Where  You  Are.  You  must  know  where  you  are  on  the  map  and  on 
the  ground  at  all  times  and  in  every  possible  way.  This  includes  knowing  where  you  are 
relative  to: 

•  Your  orientation 

•  The  direction  and  distances  to  your  objective(s) 

•  Other  landmarks  and  features 

•  Any  impassable  terrain,  the  enemy,  civilians,  and  danger  areas 

•  Both  the  advantages  and  disadvantages  presented  by  the  terrain  between  you  and 
your  objective. 

These  steps  are  accomplished  by  knowing  how  to  read  a  map,  recognize  and 
identify  landmarks;  determine  direction;  and  measure  or  estimate  distances. 

Step  2:  Plan  the  Route.  Depending  on  the  length  and  type  of  movement  to  be 
conducted,  several  factors  should  be  considered  in  selecting  a  good  route.  These  include: 

•  Time  &  Distance 
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•  Tactical  aspects  of  terrain  (called  OCOKA:  described  below) 

•  Potential  for  encountering  the  enemy 

•  Availability  of  good  checkpoints  or  identifying  landmarks 

The  plan  must  be  the  result  of  careful  map  study  and  should  address  the 
requirements  of  the  mission  and  time  available.  It  must  also  provide  for  ease  of 
movement  and  navigation. 

Step  3:  Stay  on  the  Route.  In  order  to  know  that  you  are  still  on  the  correct  route, 
you  must  be  able  to  compare  and  recognize  the  environmental  landmarks  (roads, 
buildings,  etc.)  you  encounter  in  the  VE,  as  you  move  according  to  the  plan  you 
developed  on  the  map.  There  are  usually  two  ways  to  navigate  an  environment:  dead 
reckoning  (using  a  compass)  or  terrain  association  (recognizing  various  landmarks  from 
the  map  in  their  anticipated  positions  and  sequences  as  you  pass  them).  Because  you  will 
not  have  a  compass  in  the  VE,  you  must  use  landmark  or  terrain  association  to  navigate 
in  these  VE  exercises. 

Step  4:  Recognize  the  Objective.  Your  objective  in  these  VE  missions  is  locating 
an  LZ  for  a  helicopter.  Therefore,  recognizing  an  appropriate  location  for  the  LZ  is  the 
first  step  in  map  planning.  You  will  then  need  to  employ  terrain  association  in  order  to 
recognize  landmarks  that  identify  the  LZ  area  when  the  plan  is  evaluated  in  the  VE 
exercise. 

In  MOUT  areas,  the  ranges  of  observation  and  fields  of  fire  are  reduced  by 
structures.  Targets  are  usually  briefly  exposed  at  ranges  of  100  meters  or  less.  For  this 
reason,  you  should  evaluate  all  potential  enemy  threats  within  100  meters  of  yourself  at 
all  times  and  make  your  movement  and  emplacement  plans  accordingly. 

Bounding  Overwatch 

When  moving  down  a  street,  the  dismounted  platoon  should  move  in  at  least  two 
separate  squads,  with  one  squad  moving  forward  while  the  other  squad  provides  cover. 
This  process  of  “leap-frogging”  ensures  safer  movement  through  an  urban  terrain. 
Locations  where  one  team  should  stop  while  the  other  moves  forward  are  called 
Bounding  Overwatch  (BO)  positions.  Figure  1  provides  a  diagram  of  the  movement 
pattern. 
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Figure  1.  “Leapfrog”  Movement 


Each  BO  position  should  offer  cover  and  concealment  for  the  occupying  squad  as 
well  as  open  lines  of  fire  to  cover  the  bounding  or  next  forward  moving  squad.  Example 
BO  positions  include  the  comers  of  buildings  or  beside  thick  brush  or  trees.  BO  positions 
should  not  be  more  than  30  to  50  meters  apart,  as  a  greater  separation  would  not  provide 
the  cover  necessary  for  safe  movement. 

In  some  cases,  you  will  need  to  plan  for  crossing  through  relatively  open  areas, 
such  as  streets,  alleys,  and  parks.  These  should  be  avoided  whenever  possible  as  they  are 
natural  kill  zones.  If  such  movements  are  necessary,  however,  take  extra  precautions  to 
protect  the  platoon  by  planning  for  the  most  efficient  path  (i.e.,  the  fastest)  between 
relatively  safe  areas,  such  as  near  buildings.  This  means  selecting  good  overwatch 
positions  that  cover  short  distance  movements  by  the  bounding  squad.  Note  that  this  may 
mean  taking  a  longer  route  or  setting  multiple  BO  positions  to  an  objective  in  order  to 
obtain  good  overwatch  or  a  safer  path. 

Choosing  the  best  path  depends  on  multiple  factors  related  to  the  natural 
surroundings,  obstacles,  and  potential  enemy  threats.  To  assist  in  this  process,  you  should 
analyze  the  terrain  using  the  following  military-based  criteria,  referred  to  by  the  acronym 
OCOKA. 

•  Observation  and  fields  of  fire, 

•  Cover  and  concealment, 

•  Obstacles, 

•  Key  terrain 

•  Avenues  of  approach. 

Observation  and  Fields  of  Fire.  Observation  means  seeing  the  enemy  but  not 
being  seen.  Anything  that  can  be  seen  can  be  hit.  Therefore,  a  field  of  fire  is  an  area  that  a 
weapon  or  a  group  of  weapons  can  cover  effectively  with  fire  from  a  given  position.  For 
purposes  of  this  experiment,  100  meters  is  set  as  the  operational  range  for  weapons. 
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Cover  and  Concealment.  Cover  is  either  natural  or  artificial  shelter  or  protection 
from  enemy  fire.  Always  try  to  use  covered  routes  and  seek  cover  for  each  halt  or 
overwatch  position,  no  matter  how  brief  the  halt  is. 

Concealment  is  protection  from  observation  or  surveillance,  including  concealment 
from  enemy  air  observation.  When  you  are  moving,  concealment  is  generally  secondary 
to  cover;  therefore,  select  routes  and  positions  that  protect  your  teams  and  minimize  the 
potential  for  enemy  forces  detecting  or  shooting  at  you.  Also,  ensure  that  your  squad  can 
place  covering  fire  on  potential  threats. 

Obstacles.  Obstacles  are  any  obstructions  that  stop,  delay,  or  divert  movement. 
Obstacles  can  be  natural  (rivers,  swamps,  cliffs,  or  mountains)  or  they  may  be  artificial 
(barbed  wire  entanglements,  pits,  or  buildings).  Always  consider  any  possible  obstacles 
along  your  movement  route  and,  if  possible,  try  to  keep  obstacles  between  the  enemy  and 
yourself  (see  Cover  &  Concealment  above). 

Key  Terrain.  Key  terrain  is  any  locality  or  area  that  when  seized  or  retained  offers 
a  marked  advantage.  Within  an  urban  area,  higher  buildings  may  dominate  an  area, 
offering  observation  and  fields  of  fire.  As  mentioned  before,  narrow  areas  between 
buildings  may  offer  cover  or  may  be  killing  zones  covered  by  enemy  weapons.  Moving 
through  large  open  areas  must  also  be  avoided.  You  should  always  attempt  to  identify 
and  use  key  terrain  to  your  advantage. 

Avenues  of  Approach.  These  are  access  routes.  They  may  be  the  routes  you  can 
use  to  get  to  the  enemy  or  the  routes  they  can  use  to  get  to  you.  These  include  how  you 
move  to  your  goal.  Basically,  an  identifiable  route  that  approaches  a  position  or  location 
is  an  avenue  of  approach  to  that  location.  In  urban  terrain,  these  avenues  may  be  streets 
or  open  areas  that  provide  relatively  easy  movement  combined  with  many  opportunities 
for  cover  and  concealment. 

In  the  experiment  exercises,  the  BO  is  marked  in  a  two-step  process.  First,  you 
make  a  YELLOW  circle  around  the  desired  BO  position.  Next,  you  place  a  YELLOW 
“X”  in  the  center  of  the  circle.  Figure  2  provides  an  example.  You  do  not  need  to 
indicate  which  team  uses  which  position,  as  depicted  in  Figure  1 .  You  do  need  to 
consider  all  potential  threats  to  the  moving  squad  and  position  the  BO  position 
appropriately.  In  our  exercises  you  will  not  be  allowed  to  use  building  interiors  (or 
roofs),  so  all  positions  will  be  outside. 
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Figure  2.  Bounding  Overwatch  Mark 


Landing  Zones.  Helicopters  need  a  large  amount  of  space,  free  of  debris  and 
obstacles,  to  land  safely.  Your  goal  is  to  locate  and  mark  an  appropriate  Landing  Zone 
(LZ)  for  a  single  helicopter.  Based  on  your  analysis  of  the  map,  you  will  select  a  space 
for  the  LZ,  and  then  plan  out  a  path  for  the  platoon  to  follow  when  moving  to  and 
establishing  the  LZ.  The  general  rule  for  clearance  for  an  LZ  is  an  open  space  that  is  100 
by  50  meters,  or  approximately  the  size  of  a  football  field.  The  landing  area  and  marks 
(see  below)  should  be  centered  in  this  open  space. 

The  LZ  should  not  be  on  an  uneven  slope  -  one  that  exceeds  seven  degrees,  so  the 
topography  should  be  evaluated  for  that  constraint.  A  rule  of  thumb  is  that  if  the  surface 
drops  one  meter  over  a  distance  of  10  meters,  it  slopes  too  much.  An  inspection  of  the 
map  contours  or  consideration  of  the  “lay  of  the  land”  is  used  to  make  this  decision. 

Marking  an  LZ  in  this  experiment,  as  in  the  Army,  follows  a  specific  protocol  that 
is  shown  in  Figure  3.  Normally  the  Army  marks  an  LZ  using  flags,  smoke,  or  lights  for 
night  operations.  In  all  LZ  marking,  four  circular  marks  are  arranged  in  a  “Y”  or  “T” 
pattern  (see  Figure  3)  with  the  long  stem  of  the  Y  running  in  the  direction  of  flight  a 
helicopter  should  take  when  landing  and  taking  off.  The  two  marks  at  the  top  of  the  Y  are 
14  meters  apart  (remember  that  each  full  step  in  the  VE  is  one  meter).  These  marks 
indicate  the  landing  threshold  for  the  aircraft.  From  the  top  of  the  Y  to  the  bottom  of  the 
stem  is  21  meters.  The  central  landing  mark,  where  the  helicopter  lands,  is  placed  14 
meters  from  the  top  two.  The  last  mark  is  placed  7  meters  further  down.  The  center  mark 
should  be  placed  in  the  center  of  the  clear  landing  area  (the  middle  of  the  100x50m 
field).  This  means  the  center  mark  should  be  in  the  middle  of  the  landing  area,  with  50 
meters  of  clearance  each  way  in  the  direction  of  flight  and  25  meters  clearance  to  each 
side.  Figure  4  provides  an  example  LZ  mark  on  an  enlarged,  but  correctly  colored  map. 
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Direction  of  Flight 


Figure  3.  Landing  Zone  Marks 


Observation/Fire  Post  (OFP):  Once  an  LZ  is  located  and  marked,  it  needs  to  be 
protected  from  enemy  fire.  Your  final  task  will  be  to  mark  Observation/Fire  Posts  (OFPs) 
around  the  LZ  to  indicate  where  platoon  members  can  set  up  posts  to  cover  threats  or 
approaches  to  the  LZ.  You  need  to  set  up  enough  OFPs  to  provide  complete  coverage 
around  the  LZ  (see  OCOKA,  above).  Mark  an  OFP  in  the  same  way  the  other  sites  are 
marked,  using  RED  to  distinguish  the  OFP  from  LZ  and  BO  marks.  Figures  5  and  6 
provide  examples  of  the  marks  and  positioning  for  the  OFP  sites. 

Note  that  good  OFPs  can  support  each  other  with  overlapping  fields  of  fire,  that 
means  at  least  two  positions  can  see  each  other  and  shoot  at  the  same  approaching  force. 
They  are  not  positioned  so  that  they  are  shooting  toward  each  other  in  order  to  shoot  at 
the  approaching  forces. 
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Figure  5.  Observation/Fire  Post  Mark 


Figure  6.  Example  Observation/Fire  Posts 


Map  Reading  For  Urban  Terrain 

Maps  provide  a  scaled  representation  of  environments  using  symbols  and  colors.  In 
the  MOUT  context  used  in  this  research,  the  map  is  a  topographical  representation  that  is 
1 :5000  scale.  This  means  that  one  meter  on  the  map  corresponds  to  5000  meters  on  the 
actual  terrain.  On  the  map  this  means  one  centimeter  equals  50  meters  on  the  terrain. 

The  map's  legend  contains  the  scale  and  symbols  used  on  the  map.  The  legend  will  also 
provide  the  contour  interval  (e.g.  10  meters,  the  difference  in  height  of  the  terrain 
between  each  contour  line),  and  information  about  the  markings  for  roads,  buildings, 
powerlines,  railroads,  etc. 
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Topographic  maps  use  special  symbols  and  colors  to  indicate  features  of  the 
environment.  To  facilitate  the  identification  of  features  on  a  map,  topographical  and 
cultural  information  is  printed  in  different  colors.  The  maps  you  will  use  in  the  exercises 
use  the  following  colors  and  features: 

Black  indicates  cultural  (man-made)  features  such  as  roads,  surveyed  spot 
elevations,  and  all  labels. 

Red-Brown  are  combined  to  identify  cultural  features  like  buildings. 

Blue  identifies  hydrography  or  water  features  such  as  lakes,  swamps,  rivers, 
and  drainage.  Solid  blue  indicates  lakes  or  rivers,  dotted  or  broken  lines  indicate 
occasional  water  like  flood  zones  or  intermittent  streams. 

Green  identifies  vegetation  with  military  significance,  such  as  woods, 
orchards,  and  vineyards.  Denser  green  marks  indicate  denser  vegetation.  For 
example,  dotted  green  lines  might  indicate  sparse  shrubs  and  trees,  whereas  solid 
green  areas  indicate  thick  forest  regions. 

A  contour  line  is  an  imaginary  boundary  line  that  represents  elevation,  that  is, 
vertical  distance,  above  or  below  sea  level.  Index  contour  lines  are  thicker  than  others  and 
are  marked  with  an  elevation.  Moving  from  a  smaller  to  a  larger  numbered  index  contour 
line  would  indicate  an  increase  in  elevation.  The  lines  between  the  index  contour  lines 
(normally  four)  are  thinner;  each  represents  a  change  in  elevation  of  10  meters.  Note  that 
some  variation  in  elevation  does  occur  between  contour  lines,  i.e.,  the  terrain  between 
these  lines  can  vary  in  height  to  some  degree. 

Contour  lines  are  helpful  in  showing  geographical  features  of  importance  to 
dismounted  Soldiers.  For  example,  contour  lines  that  are  evenly  spaced  and  close 
together  indicate  a  uniform,  steep  slope  of  the  terrain.  Such  a  high  point  might  be  used  as 
an  observation/fire  post.  The  closer  together  the  contour  lines,  the  steeper  the  slope;  the 
further  apart  the  contour  lines,  the  gentler  the  slope.  Furthermore,  the  direction  of  the 
slope  is  indicated  by  the  index  contour  lines.  The  contour  lines  in  Figure  7  indicate  a 
slope  upward,  from  100  meters  to  150  meters. 
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Figure  7.  Example  Contour  Lines 

The  scale  in  Figure  7  is  1 :5000,  meaning  the  3cm  on  the  map  between  the  100  and 
150  index  contour  lines  indicates  a  rise  of  50  meters  elevation  over  150  meters  of 
distance.  The  slope  between  the  100  and  150  numbers  is  quite  steep,  especially  where 
the  contour  lines  are  close  together.  Slopes  can  be  interpreted  using  boundary  estimates. 
For  example,  a  rise  of  1 00  meters  over  a  distance  of  1 00  meters  would  be  a  45  degree 
slope  (a  1:1  ratio),  50m  rise  over  100m  (a  1 :2  ratio)  is  a  27  degree  slope,  and  a  10m  over 
1 00m  (1:10  ratio)  is  a  6  degree  slope.  Other  slopes  can  be  evaluated  by  establishing  the 
ratio  and  interpolating  using  these  boundary  values. 

Since  the  LZ  has  to  be  in  a  flat  area,  an  area  with  a  1 : 1 0  ratio  or  less  is  required. 
Note  that  different  slopes  can  appear  between  contour  lines,  as  they  are  only  discrete 
measures  of  the  terrain,  and  don’t  provide  information  about  any  change  in  elevation  until 
the  change  exceeds  the  contour  interval.  On  figure  7,  an  LZ  could  only  be  placed  in  the 
flat  area  below  the  100  contour  line,  near  the  road  and  building  at  the  bottom  right  of  the 
figure. 
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Appendix  C 


Training  Test  &  Map 


7.1  Test  for  IF  Training 


Participant  #: 
Date: 


1 .  What  color  are  observation  fire  posts? 

A.  Red  B.  Green  C.  Yellow  D.  White 

2.  What  color  are  Bounding  Overwatches? 

A.  Red  B.  Green  C.  Yellow  D.  White 

3.  What  color  are  the  Landing  Zone  markers? 

A.  Red  B.  Green  C.  Yellow  D.  White 

4.  What  color  of  paint  is  used  to  erase  marks? 

A.  Red  B.  Green  C.  Yellow  D.  White 

5.  When  making  marks  in  the  environment,  you  should  be  aware  of  potential  enemy  threats 

within _ meters  of  your  position. 

A. 10  B.  100  C. 200  D.  1000 

6.  Marks  used  to  indicate  where  one  squad  should  stop  to  cover  another  squad  are  called: 

A.  Observation  Fire  Posts  B.  Bounding  Overwatches 

C.  Cover  Points  D.  Landing  Zone  Posts 

7.  On  topographical  maps,  the  colors  red  or  brown  are  used  to  indicate: 

A.  Cultural  features  like  buildings  B.  Lakes  and  streams 

C.  Roads  and  paths  D.  Vegetation  (e.g.,  trees,  bushes,  etc.) 

8.  On  topographical  maps,  the  color  black  is  used  to  indicate: 

A.  Cultural  features  like  buildings  B.  Lakes  and  streams 

C.  Roads  and  paths  D.  Vegetation  (e.g.,  trees,  bushes,  etc.) 

9.  On  topographical  maps,  the  color  green  is  used  to  indicate: 

A.  Cultural  features  like  buildings  B.  Lakes  and  streams 

C.  Roads  and  paths  D.  Vegetation  (e.g.,  trees,  bushes,  etc.) 
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10.  On  the  figure  below,  please  mark  the  appropriate  distances,  in  centimeters,  for  the 
Landing  Zone. 


Direction  of  Flight 


Left  Stem 


Right  Stem 


1 1 .  What  is  the  correct  order,  from  first  to  last,  of  marks  you  should  make  during  a  exercise? 

A.  Observation  Fire  Posts,  Bounding  Overwatches,  Landing  Zone 

B.  Bounding  Overwatches,  Landing  Zone,  Observation  Fire  Posts 

C.  Bounding  Overwatches,  Observation  Fire  Posts,  Landing  Zone 

D.  Landing  Zone,  Observation  Fire  Posts,  Bounding  Overwatches 

12.  What  is  your  primary  goal  during  the  exercises? 

A.  Protect  yourself  from  enemy  fire  B.  Detect  enemy  locations 

C.  Locate  and  mark  a  helicopter  landing  zone  D.  Shoot  all  enemy  personnel  and  tanks 

13.  The  terrain  slope  ratio  for  a  Landing  Zone  should  not  exceed  what  ratio  between  a  rise  in 
elevation  and  distance? 

A.  1:1  (100  meters  riser  over  100  meters  distance) 

B.  1 :2  (50  meters  rise  over  100  meters  distance) 

C.  1:10(10  meters  rise  over  1 00  meters  distance) 

D.  1:100  (1  meter  rise  over  100  meters  distance) 
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14.  In  the  figure  below,  which  of  the  following  statements  is  true? 

A.  Point  A  is  higher  than  point  B 

B.  Point  B  is  higher  than  point  A 

C.  Point  A  and  B  are  at  the  same  elevation 

D.  Cannot  tell  if  point  A  is  higher  or  lower  than  point  B 


15.  The  purpose  of  an  Observation/Fire  Posts  is  to 

A.  Provide  fire  coverage  to  all  areas  around  a  Landing  Zone 

B.  Provide  markers  for  helicopter  pilots 

C.  Light  up  the  area  surrounding  a  Landing  Zone 

D.  Indicate  where  teams  should  stop  while  covering  a  moving  team 


Participants  missing  three  questions  were  required  to  review  the  Training  manual  again. 
Any  subsequent  mistakes  were  grounds  for  disqualification  from  the  research. 
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Map  Planning 


Find  an  appropriate  space  for  the  LZ  and  plan  a  path  to  that  area  using  OCOKA  and  the 
movement  guidelines  presented  above  to  select  and  mark  the  BO  positions.  Lay  out  the  marks 
for  the  LZ,  and  the  OFPs  that  will  control  access  and  protect  the  area.  Mark  all  positions, 
appropriately.  A  small  ruler  will  be  furnished,  along  with  colored  markers. 


Grading  Criteria: 

Many  paths  to  the  only  available  LZ  (see  below)  are  available.  Bounding  Overwatch  rules 
provided  in  the  training  manual  should  be  applied  to  evaluate  the  overlapping  coverage  and 
OCOKA  rules  applications. 

Given  the  scale  and  configuration  of  the  buildings  on  the  map  together  with  the  contour, 
there  is  only  one  location  for  a  Landing  Zone,  the  top  left  open  area.  Given  the  scale,  the 
placement  should  be  sufficient  to  clear  the  buildings  in  that  area  and  large  enough  to  fit  the  LZ 
parameters. 

Placement  of  the  Observation/Fire  Posts  should  provide  direct  observation  of  all 
approaches  and  buildings,  within  100  meters  and  without  overlapping  an  adjacent  post. 

Violations  would  be  explained  to  the  participant.  More  than  three  major  errors  would 
require  review  of  the  appropriate  section  of  the  training  manual,  and  a  repetition  of  the  map 
analysis.  Subsequent  excessive  errors  resulted  in  disqualification  of  the  participant. 
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Appendix  D 


Virtual  Environment  Maps 


Ft.  Polk,  Shugart-Gordon  MOUT,  Scenario  1 . 


D-  1 


Ft.  Polk,  Shugart-Gordon  MOUT,  Scenario  2. 
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